Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages, which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission if necessary.

Note: Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

In addition to implementing code, there is a writeup to complete. The writeup should be completed in a separate file, which can be either a markdown file or a pdf document. There is a write up template that can be used to guide the writing process. Completing the code template and writeup template will cover all of the rubric points for this project.

The rubric contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this Ipython notebook and also discuss the results in the writeup file.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 0: Load The Data

In [1]:
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

from tqdm import tqdm
import os
from skimage.transform import resize
from random import shuffle
import pandas as pd
import cv2
from collections import Counter
from skimage import exposure
from sklearn.utils import shuffle as shuffle_X_y

data_dir = './data'
In [2]:
import pickle

# TODO: Fill this in based on where you saved the training and testing data
traffic_signs_data_dir = "%s/traffic-signs-data" % data_dir
training_file =  '%s/train.p' % traffic_signs_data_dir
validation_file = '%s/valid.p' % traffic_signs_data_dir
testing_file = '%s/test.p' % traffic_signs_data_dir
    
# Load pickled data
with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']

Step 1: Dataset Summary & Exploration

The pickled data is a dictionary with 4 key/value pairs:

  • 'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
  • 'labels' is a 1D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
  • 'sizes' is a list containing tuples, (width, height) representing the original width and height the image.
  • 'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below. Use python, numpy and/or pandas methods to calculate the data summary rather than hard coding the results. For example, the pandas shape method might be useful for calculating some of the summary results.

In [3]:
signnames_pd = pd.read_csv('signnames.csv')
signnames_pd.head()
Out[3]:
ClassId SignName
0 0 Speed limit (20km/h)
1 1 Speed limit (30km/h)
2 2 Speed limit (50km/h)
3 3 Speed limit (60km/h)
4 4 Speed limit (70km/h)
In [4]:
def signname_val_to_name(signnames_pd, val):
    return signnames_pd[signnames_pd['ClassId'] == val]['SignName'].values[0]
In [5]:
def make_label_image_dict(X, y):
    image_dict = {}
    
    for i in range(X.shape[0]):
        curr_image = X[i]
        curr_y = y[i]
        
        if curr_y in image_dict:
            image_dict[curr_y].append(curr_image)
        else:
            image_dict[curr_y] = [curr_image]
    
    return image_dict
In [6]:
def filter_images_by_label(label, X=None, y=None, label_image_dict=None):
    assert (X is not None and y is not None) or (label_image_dict is not None)
    
    filtered_image_arr = []

    if label_image_dict is None:
        label_image_dict = make_label_image_dict(X, y) 
        
    if label in label_image_dict:
        filtered_image_arr = label_image_dict[label]    
        return np.stack(filtered_image_arr)
    else:
        return None
In [7]:
def apply_to_layers(image, apply_fnc):
    image_arr = [apply_fnc(image[i]) for i in range(image.shape[0])]
    
    return np.stack(image_arr)

Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas

In [8]:
### Replace each question mark with the appropriate value. 
### Use python, pandas or numpy methods rather than hard coding the results

# TODO: Number of training examples
n_train = X_train.shape[0]

# TODO: Number of validation examples
n_validation = X_valid.shape[0]

# TODO: Number of testing examples.
n_test = X_test.shape[0]

# TODO: What's the shape of an traffic sign image?
image_shape = (X_train.shape[1], X_train.shape[2])

# TODO: How many unique classes/labels there are in the dataset.
n_classes = len(set(y_train.reshape((-1)).tolist()))

print("Number of training examples =", n_train)
print("Number of validation examples =", n_validation)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 34799
Number of validation examples = 4410
Number of testing examples = 12630
Image data shape = (32, 32)
Number of classes = 43

Include an exploratory visualization of the dataset

Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.

The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.

NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections. It can be interesting to look at the distribution of classes in the training, validation and test set. Is the distribution the same? Are there more examples of some classes than others?

In [9]:
### Data exploration visualization code goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline
In [10]:
def display_X_y_images(X, y=None, signnames_pd=None, start_idx=0, end_idx=None, 
                       columns = 5, use_gray=False, apply_fnc=None):
    if end_idx is None:
        end_idx = X.shape[0]
        
    if apply_fnc is None:
        apply_fnc = lambda image: image
        
    plt.figure(figsize=(32,18))

    num_of_images = end_idx - start_idx
    rows = num_of_images / columns + 1
    
    for i in range(start_idx, end_idx):
        image = X[i]
        
        _i = i % num_of_images
        plt.subplot(rows, columns, _i + 1)
        
        if y is None or signnames_pd is None or y[i] == -1:
            title = 'Unlabeled'
        else:
            title = "%d: %s" % (y[i],
                                signnames_pd.loc[signnames_pd['ClassId']==y[i]]['SignName'].values[0])
            
        plt.title(title)
        
        if use_gray:
            plt.imshow(apply_fnc(image), cmap="gray")
        else:
            plt.imshow(apply_fnc(image)) 
            
    plt.tight_layout()
            
    plt.show()

Sample images

Here are 30 sample images from the training set.

In [11]:
X_train, y_train = shuffle_X_y(X_train, y_train)
In [12]:
display_X_y_images(X_train, y_train, signnames_pd, start_idx=0, end_idx=30)

Distribution of Training Images by Label

In [13]:
label_cnts = Counter(y_train)

x_vals = [x for x, _ in label_cnts.items()]
y_vals = [y for _, y in label_cnts.items()]    

plt.figure(figsize=(32,18))

plt.barh(x_vals, y_vals)
plt.title("Number of traffic signs per label").set_fontsize(48)
plt.yticks(x_vals, [signname_val_to_name(signnames_pd, val) for val in x_vals], fontsize = 20)
plt.xlabel('Number of traffic signs').set_fontsize(24)
plt.show()
In [14]:
label_cnt_arr = label_cnts.most_common(43)
least_common_label, least_common_cnt = label_cnt_arr[-1]
least_common_label_name = signname_val_to_name(signnames_pd, least_common_label)

most_common_label, most_common_cnt = label_cnt_arr[0]
most_common_label_name = signname_val_to_name(signnames_pd, most_common_label)

print("Label with least number of examples: (%d)%s - %d" % (least_common_label, least_common_label_name, 
                                                            least_common_cnt))
print("Label with most number of examples: (%d)%s - %d" % (most_common_label, most_common_label_name, 
                                                            most_common_cnt))
Label with least number of examples: (37)Go straight or left - 180
Label with most number of examples: (2)Speed limit (50km/h) - 2010

We can see that distribution of the examples are imbalanced. Later, the data will be augmented to balance the examples for each label.


Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!

With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89. To meet specifications, the validation set accuracy will need to be at least 0.93. It is possible to get an even higher accuracy, but 0.93 is the minimum for a successful project submission.

There are various aspects to consider when thinking about this problem:

  • Neural network architecture (is the network over or underfitting?)
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

Pre-process the Data Set (normalization, grayscale, etc.)

Minimally, the image data should be normalized so that the data has mean zero and equal variance. For image data, (pixel - 128)/ 128 is a quick way to approximately normalize the data and can be used in this project.

Other pre-processing steps are optional. You can try different techniques to see if it improves performance.

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project.

In [15]:
### Preprocess the data here. It is required to normalize the data. Other preprocessing steps could include 
### converting to grayscale, etc.
### Feel free to use as many code cells as needed.

Augment data

For the training data, the numbers of images per label are not even. Augment the data by randomly choosing images with replacement then applying one of the following transformations:

  1. Rotate
  2. Shift
  3. Shear
In [16]:
from skimage.transform import rotate, AffineTransform, warp
from random import uniform, randrange
import cv2

def apply_random_rotate(image, min_angle=-20.0, max_angle=20.0):
    random_angle = uniform(min_angle, max_angle)
    
    return (255.0*rotate(image, random_angle)).astype(np.uint8)

def apply_random_shift(image, min_x_shift=-5, max_x_shift=5, 
                        min_y_shift=-5, max_y_shift=5):
    x_shift = randrange(min_x_shift, max_x_shift)
    y_shift = randrange(min_y_shift, max_y_shift)
    
    cols,rows = image.shape[0], image.shape[1]
    
    M = np.float32([[1,0,x_shift],[0,1,y_shift]])
    return cv2.warpAffine(image,M,(cols,rows))

def apply_random_shear(image, min_shear=-0.2, max_shear=0.2):
    shear = uniform(min_shear, max_shear)
    
    # Create Afine transform
    afine_tf = AffineTransform(shear=shear)

    # Apply transform to image data
    return warp(image, inverse_map=afine_tf)

Rotate

The range for the rotate is from -20.0 to 20.0 degrees.

In [17]:
display_X_y_images(X_train, y_train, signnames_pd, 0, 5, use_gray=True, 
                   apply_fnc=apply_random_rotate)

Shift

The image shift is +/-5 up or down and left or right.

In [18]:
display_X_y_images(X_train, y_train, signnames_pd, 0, 5, use_gray=True, 
                   apply_fnc=apply_random_shift)

Shear

The image shear is from -0.2 to 0.2 radians.

In [19]:
display_X_y_images(X_train, y_train, signnames_pd, 0, 5, use_gray=True, 
                   apply_fnc=apply_random_shear)
In [20]:
def balance_images(X, y, min_angle=-12.5, max_angle=12.5, 
                   min_x_shift=-3, max_x_shift=3, min_y_shift=-5, max_y_shift=5, 
                   min_shear=-0.2, max_shear=0.2):
    #print("###1. balance_images - min_x_shift, max_x_shift, min_y_shift, max_y_shift", 
    #                  min_x_shift, max_x_shift, min_y_shift, max_y_shift)
    label_cnts = Counter(y_train)
    
    max_cnt = max([y for _, y in label_cnts.items()])
    
    image_dict = {}
    for i in range(X.shape[0]):
        curr_image = X[i]
        curr_y = y[i]
        
        if curr_y in image_dict:
            image_dict[curr_y].append(curr_image)
        else:
            image_dict[curr_y] = [curr_image]
            
    for label, image_arr in image_dict.items():
        initial_cnt = len(image_arr)
        for i in range(max_cnt-initial_cnt):
            idx = randrange(0, initial_cnt)
            curr_image = image_arr[idx]
                
            rnd = randrange(0,3)    
            if rnd == 0:
                alt_image = apply_random_rotate(curr_image, min_angle, max_angle)  
            elif rnd == 1:
                alt_image = apply_random_shift(curr_image, min_x_shift, max_x_shift, 
                                               min_y_shift, max_y_shift)                 
            else:
                alt_image = apply_random_shear(curr_image, min_shear, max_shear)
                
            image_dict[label].append(alt_image)
            
    balanced_label_arr = []
    balanced_image_arr = []
    for label, image_arr in image_dict.items():
        balanced_label_arr += [label] * len(image_arr)
        balanced_image_arr += image_arr
        
    balanced_labels = np.array(balanced_label_arr).reshape((-1))
    balanced_images = np.stack(balanced_image_arr)
    
    return balanced_images, balanced_labels
In [21]:
X_train_balanced, y_train_balanced = balance_images(X_train, y_train, 
                                                    min_angle=-20.0, max_angle=20.0, 
                                                    min_x_shift=-5, max_x_shift=5, 
                                                    min_y_shift=-5, max_y_shift=5, 
                                                    min_shear=-0.2, max_shear=0.2)

X_train_balanced, y_train_balanced = shuffle_X_y(X_train_balanced, y_train_balanced)

Grayscale images with Histogram Equalization

In [22]:
def apply_histogram_equalization(image):
    gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    equalized_image = exposure.equalize_adapthist(gray_image)
    
    return (255.0*equalized_image).astype(np.uint8)

Here is a version of the same images in grayscale. Histogram Equalization has been applied to sharpen the images.

In [23]:
display_X_y_images(X_train, y_train, signnames_pd, start_idx=20, end_idx=25, 
                   use_gray=True, apply_fnc=apply_histogram_equalization)
In [24]:
def get_color_layers(image):
    temp_image = image.copy()

    layers_arr = [temp_image[:,:,i] for i in range(temp_image.shape[2])]

    he_image = apply_histogram_equalization(temp_image)
    layers_arr.append(he_image)
    
    color_layers = np.stack(layers_arr, axis=2)
    
    assert color_layers.shape[2] == 4, color_layers.shape
    
    return color_layers
In [25]:
def preprocess_images(images):
    #temp_images = images.copy()
    
    color_layers_images = np.stack([get_color_layers(images[i]) 
                                    for i in tqdm(range(images.shape[0]))]).astype(np.float32)
            
    assert color_layers_images.shape[0] == images.shape[0], (color_layers_images.shape[0],
                                                            images.shape[0])
        
    # normalize data
    for i in tqdm(range(color_layers_images.shape[3])):
        curr_layer = color_layers_images[:, :, :, i]
        min_val = curr_layer.min()
        max_val = curr_layer.max()
        mean_val = curr_layer.mean()
            
        val_range = max_val - min_val + 1.0**-9
        
        #masked_images[:, :, :, i] = 0.5*(curr_layer-mean_val)/val_range    
        color_layers_images[:, :, :, i] = (curr_layer-mean_val)/val_range

    return color_layers_images
In [26]:
import multiprocessing as mp

class MP(object):
    def __init__(self, worker, input_splitter, output_joiner, num_of_workers=3):
        self.worker = worker
        self.input_splitter = input_splitter
        self.output_joiner = output_joiner
        self.num_of_workers = num_of_workers
        
    def split_input(self, input, n):
        return [{'idx':i, 'data':input_part} for i, input_part in enumerate(self.input_splitter(input, n))]
    
    def run(self, input):
        pool = mp.Pool()
        
        split_input = self.split_input(input, self.num_of_workers)
    
        output = pool.map(self.worker, split_input)
        
        return self.output_joiner(output, self.num_of_workers)
In [27]:
def preprocess_images_worker(val):
    idx, images = val['idx'], val['data']
  
    result = preprocess_images(images)
        
    return {'idx': idx, 'result': result}
    
def input_splitter(images, n):
    return [x for x in np.array_split(images, n)]
        
def output_joiner(output, n):
    output_dict = {x['idx']: x['result'] for x in output}
        
    output_arr = [output_dict[i] for i in range(n)]
        
    return np.concatenate(output_arr, axis=0)

def preprocess_images_mt(images, num_of_workers):     
    preprocess_images_mp = MP(preprocess_images_worker, input_splitter, output_joiner, num_of_workers)
    return preprocess_images_mp.run(images)
In [28]:
def preprocess_X_y(X, y, data_dir, type_name, n=8):
    X_preprocessed_file = '%s/X_%s_preprocessed' % (data_dir, type_name)
    y_preprocessed_file = '%s/y_%s_preprocessed' % (data_dir, type_name)
    if os.path.exists('%s.npy' % X_preprocessed_file) \
        and os.path.exists('%s.npy' % y_preprocessed_file):
        X_preprocessed = np.load('%s.npy' % X_preprocessed_file)
        y_preprocessed = np.load('%s.npy' % y_preprocessed_file)
    else:
        X_preprocessed = preprocess_images_mt(X, n)
        y_preprocessed = y
        np.save(X_preprocessed_file, X_preprocessed)
        np.save(y_preprocessed_file, y_preprocessed)

    return X_preprocessed, y_preprocessed
In [29]:
X_train_preprocessed, y_train_preprocessed = preprocess_X_y(X_train, y_train, 
                                                             data_dir, 'train',
                                                             n=8)
In [30]:
X_valid_preprocessed,  y_valid_preprocessed = preprocess_X_y(X_valid, y_valid, 
                                                             data_dir, 'valid',
                                                             n=8)
In [31]:
X_test_preprocessed,  y_test_preprocessed = preprocess_X_y(X_test, y_test, 
                                                           data_dir, 'test',
                                                             n=8)

Model Architecture

In [32]:
### Define your architecture here.
### Feel free to use as many code cells as needed.
In [33]:
import tensorflow as tf
from tensorflow.contrib.layers import flatten
In [34]:
def leaky_relu(x, alpha, name):
    return tf.maximum(alpha * x, x, name) 

def conv_layer(x, num_input_channels, num_output_channels, 
               mu, sigma, window=(5,5)):
    window_x,window_y = window
    conv_W = tf.Variable(tf.truncated_normal(shape=(window_x, window_y, 
                                                    num_input_channels, num_output_channels), 
                                                     mean = mu, stddev = sigma))
    conv_b = tf.Variable(tf.zeros(num_output_channels))
    return tf.nn.conv2d(x, conv_W, strides=[1, 1, 1, 1], padding='SAME') + conv_b, conv_W

def fully_connected(x, shape_in, shape_out, mu = 0, sigma = 0.1):
    fc_W  = tf.Variable(tf.truncated_normal(shape=(shape_in, shape_out), 
                                            mean = mu, stddev = sigma))
    fc_b  = tf.Variable(tf.zeros(shape_out))
    return tf.matmul(x, fc_W) + fc_b, fc_W
In [35]:
def model(x, channel_depth, mu = 0, sigma = 1e-2, alpha=0.05): 
    ### Layer 1: Convolutional. Input = 32x32x10. Output = 32x32x16.
    conv1, conv1_W = conv_layer(x, channel_depth, 16, mu, sigma)
    print(conv1.get_shape())

    # Leaky ReLU
    conv1_activaton = leaky_relu(conv1, alpha, 'conv1_activaton')    

    # Pooling. Input = 32x32x16. Output = 16x16x16.
    conv1 = tf.nn.max_pool(conv1_activaton, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
    print(conv1.get_shape())

    ### Layer 2: Convolutional. Output = 16x16x32.
    conv2, conv2_W = conv_layer(conv1, 16, 32, mu, sigma)
    print(conv2.get_shape())

    # Leaky ReLU
    conv2_activaton = leaky_relu(conv2, alpha, 'conv2_activaton')  

    # Pooling. Input = 16x16x32. Output = 8x8x32.
    conv2 = tf.nn.max_pool(conv2_activaton, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
    print(conv2.get_shape())

    ### Layer 3: Convolutional. Output = 8x8x64.
    conv3, conv3_W = conv_layer(conv2, 32, 64, mu, sigma)
    print(conv3.get_shape())

    # Leaky ReLU
    conv3_activaton = leaky_relu(conv3, alpha, 'conv3_activaton')  

    # Pooling. Input = 8x8x64. Output = 4x4x64.
    conv3 = tf.nn.max_pool(conv3_activaton, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME') 
    print(conv3.get_shape())

    # Flatten. Input = 4x4x64. Output = 1024.
    fc0   = flatten(conv3)

    ### Layer 3: Fully Connected. Input = 1024. Output = 512.
    fc1, fc1_W = fully_connected(fc0, 1024, 512, mu, sigma)

    # Leaky ReLU
    fc1 = leaky_relu(fc1, alpha, 'fc1_activation')  
    
    # dropout
    keep_prob = tf.placeholder(tf.float32)
    fc1 = tf.nn.dropout(fc1, keep_prob)

    ### Layer 4: Fully Connected. Input = 512. Output = 256.
    fc2, fc2_W = fully_connected(fc1, 512, 256, mu, sigma)

    # Leaky ReLU
    fc2 = leaky_relu(fc2, alpha, 'fc2_activation')   
    
    # dropout
    fc2 = tf.nn.dropout(fc2, keep_prob)    

    # Layer 5: Fully Connected. Input = 256. Output = 43.
    logits, fc3_W = fully_connected(fc2, 256, 43, mu, sigma)

    return (logits, keep_prob, [conv1_W, conv2_W, conv3_W, fc1_W, fc2_W, fc3_W],
            [conv1_activaton, conv2_activaton, conv3_activaton])
In [36]:
channel_depth = 4
x = tf.placeholder(tf.float32, (None, 32, 32, channel_depth))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)
In [37]:
rate = 1e-3       # learning rate
alpha = 1e-05     # slope for negative input values for leaky ReLU's
sigma = 1e-2      # std for initializing random weights
beta = 1e-4       # multiplier for L2 regularization

logits, keep_prob, weights, activations = model(x, channel_depth, sigma=sigma, alpha=alpha)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)

loss_operation = tf.reduce_mean(cross_entropy)

# Loss function using L2 Regularization
regularizer = None
for weight in weights:
    if regularizer is None:
        regularizer = tf.nn.l2_loss(weight)
    else:
        regularizer = regularizer + tf.nn.l2_loss(weight)
loss_operation = tf.reduce_mean(loss_operation + beta * regularizer)

optimizer = tf.train.AdamOptimizer(learning_rate = rate)
training_operation = optimizer.minimize(loss_operation)
(?, 32, 32, 16)
(?, 16, 16, 16)
(?, 16, 16, 32)
(?, 8, 8, 32)
(?, 8, 8, 64)
(?, 4, 4, 64)

Train, Validate and Test the Model

A validation set can be used to assess how well the model is performing. A low accuracy on the training and validation sets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.

In [38]:
### Train your model here.
### Calculate and report the accuracy on the training and validation set.
### Once a final model architecture is selected, 
### the accuracy on the test set should be calculated and reported as well.
### Feel free to use as many code cells as needed.
In [39]:
EPOCHS = 100
BATCH_SIZE = 128
In [40]:
logits_argmax = tf.argmax(logits, 1, name='logits_argmax')
one_hot_y_argmax = tf.argmax(one_hot_y, 1, name='one_hot_y_argmax')
correct_prediction = tf.equal(logits_argmax, one_hot_y_argmax, name='correct_prediction')
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y, keep_prob: 1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples
In [41]:
def train_model(epochs, batch_size, X_train_preprocessed, y_train, 
                X_valid_preprocessed, y_valid, model_name, keep_prob_val = 0.5,
                max_to_keep=0):
    saver = tf.train.Saver(max_to_keep=max_to_keep)
    model_dir = '%s/models/%s' % (data_dir, model_name)
    os.makedirs(model_dir, exist_ok=True)
    
    validation_accuracy_arr = []
    
    best_validation_accuracy_epoch = 0
    best_validation_accuracy = 0.0

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        num_examples = len(X_train_preprocessed)
    
        print("Training...")
        print()
        for i in range(epochs):
            X_train_preprocessed, y_train = shuffle_X_y(X_train_preprocessed, y_train)
        
            for offset in range(0, num_examples, batch_size):
                end = offset + BATCH_SIZE
                batch_x, batch_y = (X_train_preprocessed[offset:end], 
                                    y_train[offset:end])
                sess.run(training_operation, feed_dict={x: batch_x, y: batch_y, 
                                                        keep_prob: keep_prob_val})
        
            validation_accuracy = evaluate(X_valid_preprocessed, y_valid)
            validation_accuracy_arr.append(validation_accuracy)
            print("EPOCH {} ...".format(i+1))
            print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        
            if best_validation_accuracy < validation_accuracy:
                best_validation_accuracy = validation_accuracy
                best_validation_accuracy_epoch = i+1
                saver.save(sess, '%s/%s' % (model_dir, model_name))
                print("Model saved")
            
            print()
            
    print("Best model - epoch: %d, best validation accuracy: %.3f" % 
          (best_validation_accuracy_epoch, best_validation_accuracy))
    
    return validation_accuracy_arr
In [42]:
def display_test_accuracy(X_test_preprocessed, y_test, model_name):
    model_dir = '%s/models/%s' % (data_dir, model_name)
    saver = tf.train.Saver()

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        saver.restore(sess, tf.train.latest_checkpoint(model_dir))

        test_accuracy = evaluate(X_test_preprocessed, y_test)
        print("Test Accuracy = {:.3f}".format(test_accuracy))
In [43]:
def predict_model(X, model_name, batch_size):
    if X.shape[0] < batch_size:
        batch_size = X.shape[0]
        
    model_dir = '%s/models/%s' % (data_dir, model_name)
    saver = tf.train.Saver()
    
    logits_argmax_arr = []
    
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())  
        saver.restore(sess, tf.train.latest_checkpoint(model_dir))
        
        start_idx = 0
        end_idx = 0
        while start_idx < X.shape[0]:
            end_idx += batch_size
            end_idx = min(end_idx, X.shape[0])
            logits_argmax_arr.append(sess.run(logits_argmax, 
                                              feed_dict={x: X[start_idx:end_idx], 
                                                         keep_prob: 1.0}))
            start_idx += batch_size
        
    result = np.concatenate(logits_argmax_arr, axis=0)
    
    assert result.shape[0] == X.shape[0]
    
    return result
In [44]:
def predict_ensemble_models(X, model_prefix, n, batch_size):
    results = []
    for i in tqdm(range(n)):
        model_name = '%s_%d' % (model_prefix, i)
        
        results.append(predict_model(X, model_name, batch_size=batch_size))
    
    return results

Single model

In [54]:
model_name = 'single_model' 
print("###%s" % model_name)
validation_accuracy_arr = train_model(EPOCHS, BATCH_SIZE, 
                                      X_train_preprocessed, y_train_preprocessed, 
                                      X_valid_preprocessed, y_valid, 
                                      model_name, keep_prob_val = 0.5, max_to_keep=0)    
    
display_test_accuracy(X_test_preprocessed, y_test, model_name)
###single_model
Training...

EPOCH 1 ...
Validation Accuracy = 0.120
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.530
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.764
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.863
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.912
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.929
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.942
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.941

EPOCH 9 ...
Validation Accuracy = 0.953
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 10 ...
Validation Accuracy = 0.954
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 11 ...
Validation Accuracy = 0.939

EPOCH 12 ...
Validation Accuracy = 0.953

EPOCH 13 ...
Validation Accuracy = 0.951

EPOCH 14 ...
Validation Accuracy = 0.953

EPOCH 15 ...
Validation Accuracy = 0.956
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 16 ...
Validation Accuracy = 0.956
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 17 ...
Validation Accuracy = 0.944

EPOCH 18 ...
Validation Accuracy = 0.953

EPOCH 19 ...
Validation Accuracy = 0.960
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 20 ...
Validation Accuracy = 0.965
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 21 ...
Validation Accuracy = 0.960

EPOCH 22 ...
Validation Accuracy = 0.969
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 23 ...
Validation Accuracy = 0.965

EPOCH 24 ...
Validation Accuracy = 0.957

EPOCH 25 ...
Validation Accuracy = 0.964

EPOCH 26 ...
Validation Accuracy = 0.962

EPOCH 27 ...
Validation Accuracy = 0.967

EPOCH 28 ...
Validation Accuracy = 0.969

EPOCH 29 ...
Validation Accuracy = 0.963

EPOCH 30 ...
Validation Accuracy = 0.966

EPOCH 31 ...
Validation Accuracy = 0.969

EPOCH 32 ...
Validation Accuracy = 0.972
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 33 ...
Validation Accuracy = 0.972

EPOCH 34 ...
Validation Accuracy = 0.971

EPOCH 35 ...
Validation Accuracy = 0.965

EPOCH 36 ...
Validation Accuracy = 0.970

EPOCH 37 ...
Validation Accuracy = 0.969

EPOCH 38 ...
Validation Accuracy = 0.964

EPOCH 39 ...
Validation Accuracy = 0.971

EPOCH 40 ...
Validation Accuracy = 0.971

EPOCH 41 ...
Validation Accuracy = 0.970

EPOCH 42 ...
Validation Accuracy = 0.977
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 43 ...
Validation Accuracy = 0.976

EPOCH 44 ...
Validation Accuracy = 0.969

EPOCH 45 ...
Validation Accuracy = 0.963

EPOCH 46 ...
Validation Accuracy = 0.977

EPOCH 47 ...
Validation Accuracy = 0.978
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 48 ...
Validation Accuracy = 0.971

EPOCH 49 ...
Validation Accuracy = 0.973

EPOCH 50 ...
Validation Accuracy = 0.971

EPOCH 51 ...
Validation Accuracy = 0.973

EPOCH 52 ...
Validation Accuracy = 0.971

EPOCH 53 ...
Validation Accuracy = 0.973

EPOCH 54 ...
Validation Accuracy = 0.971

EPOCH 55 ...
Validation Accuracy = 0.966

EPOCH 56 ...
Validation Accuracy = 0.974

EPOCH 57 ...
Validation Accuracy = 0.972

EPOCH 58 ...
Validation Accuracy = 0.974

EPOCH 59 ...
Validation Accuracy = 0.976

EPOCH 60 ...
Validation Accuracy = 0.967

EPOCH 61 ...
Validation Accuracy = 0.980
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 62 ...
Validation Accuracy = 0.978

EPOCH 63 ...
Validation Accuracy = 0.980

EPOCH 64 ...
Validation Accuracy = 0.973

EPOCH 65 ...
Validation Accuracy = 0.972

EPOCH 66 ...
Validation Accuracy = 0.982
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 67 ...
Validation Accuracy = 0.977

EPOCH 68 ...
Validation Accuracy = 0.975

EPOCH 69 ...
Validation Accuracy = 0.972

EPOCH 70 ...
Validation Accuracy = 0.971

EPOCH 71 ...
Validation Accuracy = 0.979

EPOCH 72 ...
Validation Accuracy = 0.974

EPOCH 73 ...
Validation Accuracy = 0.981

EPOCH 74 ...
Validation Accuracy = 0.981

EPOCH 75 ...
Validation Accuracy = 0.979

EPOCH 76 ...
Validation Accuracy = 0.981

EPOCH 77 ...
Validation Accuracy = 0.980

EPOCH 78 ...
Validation Accuracy = 0.966

EPOCH 79 ...
Validation Accuracy = 0.980

EPOCH 80 ...
Validation Accuracy = 0.981

EPOCH 81 ...
Validation Accuracy = 0.978

EPOCH 82 ...
Validation Accuracy = 0.975

EPOCH 83 ...
Validation Accuracy = 0.977

EPOCH 84 ...
Validation Accuracy = 0.980

EPOCH 85 ...
Validation Accuracy = 0.975

EPOCH 86 ...
Validation Accuracy = 0.980

EPOCH 87 ...
Validation Accuracy = 0.980

EPOCH 88 ...
Validation Accuracy = 0.979

EPOCH 89 ...
Validation Accuracy = 0.980

EPOCH 90 ...
Validation Accuracy = 0.978

EPOCH 91 ...
Validation Accuracy = 0.979

EPOCH 92 ...
Validation Accuracy = 0.982

EPOCH 93 ...
Validation Accuracy = 0.983
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 94 ...
Validation Accuracy = 0.985
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 95 ...
Validation Accuracy = 0.984

EPOCH 96 ...
Validation Accuracy = 0.977

EPOCH 97 ...
Validation Accuracy = 0.982

EPOCH 98 ...
Validation Accuracy = 0.980

EPOCH 99 ...
Validation Accuracy = 0.985
INFO:tensorflow:./data/models/single_model/single_model is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 100 ...
Validation Accuracy = 0.980

Best model - epoch: 99, best validation accuracy: 0.985
Test Accuracy = 0.966
In [55]:
def get_individual_accuracy(y, y_hat):
    image_cnt = y.shape[0]

    correct_cnt = 0
    
    for i in range(image_cnt):
        if y[i] == y_hat[i]:
            correct_cnt += 1
        
    return float(correct_cnt)/image_cnt    
In [56]:
def display_single_accuracy_barplot(train_accuracy, valid_accuracy, test_accuracy, title):
    
    n_groups = 3

    # create plot
    fig, ax = plt.subplots(figsize=(5, 5))
    index = np.arange(n_groups)
    bar_width = 0.25
    opacity = 0.8
 
    rects1 = plt.bar(index, [train_accuracy, valid_accuracy, test_accuracy], bar_width,
                     alpha=opacity,
                     color=['#B3B3B3', '#1F1A4F', '#C82027'],
                     label='Train')
 
    plt.xlabel('Models').set_fontsize(12)
    plt.ylabel('Accuracies').set_fontsize(12)
    plt.title(title).set_fontsize(18)
    plt.xticks(index, ['train', 'valid', 'test'])
    plt.legend(fontsize='medium')

    plt.show()

Accuracy

In [57]:
# Train Accuracy
single_y_train_hat = predict_model(X_train_preprocessed, 'single_model', batch_size=BATCH_SIZE)

single_train_accuracy = get_individual_accuracy(y_train_preprocessed, single_y_train_hat)

# Validation Accuracy
single_y_valid_hat = predict_model(X_valid_preprocessed, 'single_model', batch_size=BATCH_SIZE)

single_valid_accuracy = get_individual_accuracy(y_valid_preprocessed, single_y_valid_hat)

# Test Accuracy
single_y_test_hat = predict_model(X_test_preprocessed, 'single_model', batch_size=BATCH_SIZE)

single_test_accuracy = get_individual_accuracy(y_test_preprocessed, single_y_test_hat)
In [65]:
def display_accuracy_table(train_accuracy, valid_accuracy, test_accuracy):
    print("Type\tAccuracy")
    print("================")
    print("train\t%.3f" % train_accuracy)
    print("valid\t%.3f" % valid_accuracy)
    print("test\t%.3f" % test_accuracy)
In [66]:
display_accuracy_table(single_train_accuracy, single_valid_accuracy, single_test_accuracy)
Type	Accuracy
================
train	1.000
valid	0.985
test	0.966
In [67]:
display_single_accuracy_barplot(single_train_accuracy, single_valid_accuracy, 
                                single_test_accuracy, 'Single Model Accuracies')
In [68]:
def pad_str(text_str, str_len, pad_char=' '):
    while len(text_str) < str_len:
        text_str += pad_char 
        
    return text_str
In [69]:
def display_incorrect_labels(y, y_hat):
    bad_labels = []
    for i in range(y.shape[0]):
        if y[i] != y_hat[i]:
            bad_labels.append(y[i])
        
    bad_label_cnt_dict = Counter(bad_labels)

    print("#     Label                                              Count")
    print("==============================================================")
    for label, count in bad_label_cnt_dict.most_common(10):
        
        row_str = "%d" % label
        row_str = pad_str(row_str, 6)
            
        row_str += signname_val_to_name(signnames_pd, label)
        
        row_str = pad_str(row_str, 60)      
        
        row_str += "%d" % count
        
        print(row_str)

Incorrect Labels

Validation

In [70]:
display_incorrect_labels(y_valid_preprocessed, single_y_valid_hat)
#     Label                                              Count
==============================================================
21    Double curve                                          10
20    Dangerous curve to the right                          6
24    Road narrows on the right                             6
1     Speed limit (30km/h)                                  5
8     Speed limit (120km/h)                                 4
5     Speed limit (80km/h)                                  3
16    Vehicles over 3.5 metric tons prohibited              3
18    General caution                                       3
41    End of no passing                                     3
0     Speed limit (20km/h)                                  2

Test

In [71]:
display_incorrect_labels(y_test_preprocessed, single_y_test_hat)
#     Label                                              Count
==============================================================
18    General caution                                       60
22    Bumpy road                                            30
30    Beware of ice/snow                                    27
6     End of speed limit (80km/h)                           22
8     Speed limit (120km/h)                                 19
26    Traffic signals                                       19
27    Pedestrians                                           19
36    Go straight or right                                  19
3     Speed limit (60km/h)                                  18
11    Right-of-way at the next intersection                 18

Ensemble model

In [72]:
n = 10
batch_perc = 0.9
models_validation_accuracy_arr = []

for i in range(n):
    sample_len = int(batch_perc*X_train_preprocessed.shape[0])
    X_train_preprocessed, y_train_preprocessed = shuffle_X_y(X_train_preprocessed, y_train_preprocessed)      
    _X_train_preprocessed = X_train_preprocessed[:sample_len]
    _y_train_preprocessed = y_train_preprocessed[:sample_len]
    
    model_name = 'model_%d' % i
    print("###%s" % model_name)
    validation_accuracy_arr = train_model(EPOCHS, BATCH_SIZE, 
                                          _X_train_preprocessed, _y_train_preprocessed, 
                                          X_valid_preprocessed, y_valid, 
                                          model_name, keep_prob_val = 0.5, max_to_keep=0) 
    models_validation_accuracy_arr.append(validation_accuracy_arr)
    
    display_test_accuracy(X_test_preprocessed, y_test, model_name)
###model_0
Training...

EPOCH 1 ...
Validation Accuracy = 0.100
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.411
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.653
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.802
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.876
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.909
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.925
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.937
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.942
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 10 ...
Validation Accuracy = 0.937

EPOCH 11 ...
Validation Accuracy = 0.937

EPOCH 12 ...
Validation Accuracy = 0.945
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 13 ...
Validation Accuracy = 0.952
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 14 ...
Validation Accuracy = 0.942

EPOCH 15 ...
Validation Accuracy = 0.943

EPOCH 16 ...
Validation Accuracy = 0.948

EPOCH 17 ...
Validation Accuracy = 0.951

EPOCH 18 ...
Validation Accuracy = 0.955
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 19 ...
Validation Accuracy = 0.955

EPOCH 20 ...
Validation Accuracy = 0.955

EPOCH 21 ...
Validation Accuracy = 0.948

EPOCH 22 ...
Validation Accuracy = 0.962
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 23 ...
Validation Accuracy = 0.963
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 24 ...
Validation Accuracy = 0.958

EPOCH 25 ...
Validation Accuracy = 0.960

EPOCH 26 ...
Validation Accuracy = 0.965
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 27 ...
Validation Accuracy = 0.954

EPOCH 28 ...
Validation Accuracy = 0.962

EPOCH 29 ...
Validation Accuracy = 0.961

EPOCH 30 ...
Validation Accuracy = 0.959

EPOCH 31 ...
Validation Accuracy = 0.967
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 32 ...
Validation Accuracy = 0.958

EPOCH 33 ...
Validation Accuracy = 0.953

EPOCH 34 ...
Validation Accuracy = 0.965

EPOCH 35 ...
Validation Accuracy = 0.956

EPOCH 36 ...
Validation Accuracy = 0.969
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 37 ...
Validation Accuracy = 0.966

EPOCH 38 ...
Validation Accuracy = 0.969

EPOCH 39 ...
Validation Accuracy = 0.962

EPOCH 40 ...
Validation Accuracy = 0.965

EPOCH 41 ...
Validation Accuracy = 0.965

EPOCH 42 ...
Validation Accuracy = 0.966

EPOCH 43 ...
Validation Accuracy = 0.958

EPOCH 44 ...
Validation Accuracy = 0.964

EPOCH 45 ...
Validation Accuracy = 0.965

EPOCH 46 ...
Validation Accuracy = 0.965

EPOCH 47 ...
Validation Accuracy = 0.967

EPOCH 48 ...
Validation Accuracy = 0.964

EPOCH 49 ...
Validation Accuracy = 0.956

EPOCH 50 ...
Validation Accuracy = 0.964

EPOCH 51 ...
Validation Accuracy = 0.966

EPOCH 52 ...
Validation Accuracy = 0.959

EPOCH 53 ...
Validation Accuracy = 0.961

EPOCH 54 ...
Validation Accuracy = 0.963

EPOCH 55 ...
Validation Accuracy = 0.959

EPOCH 56 ...
Validation Accuracy = 0.969

EPOCH 57 ...
Validation Accuracy = 0.970
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 58 ...
Validation Accuracy = 0.965

EPOCH 59 ...
Validation Accuracy = 0.970

EPOCH 60 ...
Validation Accuracy = 0.969

EPOCH 61 ...
Validation Accuracy = 0.966

EPOCH 62 ...
Validation Accuracy = 0.966

EPOCH 63 ...
Validation Accuracy = 0.971
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 64 ...
Validation Accuracy = 0.971

EPOCH 65 ...
Validation Accuracy = 0.969

EPOCH 66 ...
Validation Accuracy = 0.974
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 67 ...
Validation Accuracy = 0.972

EPOCH 68 ...
Validation Accuracy = 0.973

EPOCH 69 ...
Validation Accuracy = 0.970

EPOCH 70 ...
Validation Accuracy = 0.971

EPOCH 71 ...
Validation Accuracy = 0.962

EPOCH 72 ...
Validation Accuracy = 0.976
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 73 ...
Validation Accuracy = 0.971

EPOCH 74 ...
Validation Accuracy = 0.969

EPOCH 75 ...
Validation Accuracy = 0.962

EPOCH 76 ...
Validation Accuracy = 0.974

EPOCH 77 ...
Validation Accuracy = 0.969

EPOCH 78 ...
Validation Accuracy = 0.962

EPOCH 79 ...
Validation Accuracy = 0.958

EPOCH 80 ...
Validation Accuracy = 0.967

EPOCH 81 ...
Validation Accuracy = 0.973

EPOCH 82 ...
Validation Accuracy = 0.973

EPOCH 83 ...
Validation Accuracy = 0.978
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 84 ...
Validation Accuracy = 0.970

EPOCH 85 ...
Validation Accuracy = 0.976

EPOCH 86 ...
Validation Accuracy = 0.974

EPOCH 87 ...
Validation Accuracy = 0.979
INFO:tensorflow:./data/models/model_0/model_0 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 88 ...
Validation Accuracy = 0.977

EPOCH 89 ...
Validation Accuracy = 0.978

EPOCH 90 ...
Validation Accuracy = 0.975

EPOCH 91 ...
Validation Accuracy = 0.972

EPOCH 92 ...
Validation Accuracy = 0.973

EPOCH 93 ...
Validation Accuracy = 0.977

EPOCH 94 ...
Validation Accuracy = 0.978

EPOCH 95 ...
Validation Accuracy = 0.975

EPOCH 96 ...
Validation Accuracy = 0.968

EPOCH 97 ...
Validation Accuracy = 0.976

EPOCH 98 ...
Validation Accuracy = 0.972

EPOCH 99 ...
Validation Accuracy = 0.968

EPOCH 100 ...
Validation Accuracy = 0.975

Best model - epoch: 87, best validation accuracy: 0.979
Test Accuracy = 0.962
###model_1
Training...

EPOCH 1 ...
Validation Accuracy = 0.091
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.468
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.714
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.818
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.895
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.919
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.931
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.941
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.947
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 10 ...
Validation Accuracy = 0.948
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 11 ...
Validation Accuracy = 0.952
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 12 ...
Validation Accuracy = 0.949

EPOCH 13 ...
Validation Accuracy = 0.959
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 14 ...
Validation Accuracy = 0.944

EPOCH 15 ...
Validation Accuracy = 0.955

EPOCH 16 ...
Validation Accuracy = 0.954

EPOCH 17 ...
Validation Accuracy = 0.961
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 18 ...
Validation Accuracy = 0.952

EPOCH 19 ...
Validation Accuracy = 0.961
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 20 ...
Validation Accuracy = 0.951

EPOCH 21 ...
Validation Accuracy = 0.953

EPOCH 22 ...
Validation Accuracy = 0.970
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 23 ...
Validation Accuracy = 0.967

EPOCH 24 ...
Validation Accuracy = 0.959

EPOCH 25 ...
Validation Accuracy = 0.970
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 26 ...
Validation Accuracy = 0.963

EPOCH 27 ...
Validation Accuracy = 0.965

EPOCH 28 ...
Validation Accuracy = 0.963

EPOCH 29 ...
Validation Accuracy = 0.965

EPOCH 30 ...
Validation Accuracy = 0.965

EPOCH 31 ...
Validation Accuracy = 0.963

EPOCH 32 ...
Validation Accuracy = 0.962

EPOCH 33 ...
Validation Accuracy = 0.957

EPOCH 34 ...
Validation Accuracy = 0.955

EPOCH 35 ...
Validation Accuracy = 0.963

EPOCH 36 ...
Validation Accuracy = 0.957

EPOCH 37 ...
Validation Accuracy = 0.965

EPOCH 38 ...
Validation Accuracy = 0.964

EPOCH 39 ...
Validation Accuracy = 0.967

EPOCH 40 ...
Validation Accuracy = 0.958

EPOCH 41 ...
Validation Accuracy = 0.970

EPOCH 42 ...
Validation Accuracy = 0.965

EPOCH 43 ...
Validation Accuracy = 0.973
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 44 ...
Validation Accuracy = 0.973

EPOCH 45 ...
Validation Accuracy = 0.967

EPOCH 46 ...
Validation Accuracy = 0.969

EPOCH 47 ...
Validation Accuracy = 0.968

EPOCH 48 ...
Validation Accuracy = 0.963

EPOCH 49 ...
Validation Accuracy = 0.971

EPOCH 50 ...
Validation Accuracy = 0.975
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 51 ...
Validation Accuracy = 0.972

EPOCH 52 ...
Validation Accuracy = 0.966

EPOCH 53 ...
Validation Accuracy = 0.967

EPOCH 54 ...
Validation Accuracy = 0.970

EPOCH 55 ...
Validation Accuracy = 0.969

EPOCH 56 ...
Validation Accuracy = 0.974

EPOCH 57 ...
Validation Accuracy = 0.973

EPOCH 58 ...
Validation Accuracy = 0.973

EPOCH 59 ...
Validation Accuracy = 0.972

EPOCH 60 ...
Validation Accuracy = 0.968

EPOCH 61 ...
Validation Accuracy = 0.974

EPOCH 62 ...
Validation Accuracy = 0.970

EPOCH 63 ...
Validation Accuracy = 0.972

EPOCH 64 ...
Validation Accuracy = 0.975

EPOCH 65 ...
Validation Accuracy = 0.972

EPOCH 66 ...
Validation Accuracy = 0.973

EPOCH 67 ...
Validation Accuracy = 0.975
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 68 ...
Validation Accuracy = 0.976
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 69 ...
Validation Accuracy = 0.977
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 70 ...
Validation Accuracy = 0.976

EPOCH 71 ...
Validation Accuracy = 0.976

EPOCH 72 ...
Validation Accuracy = 0.960

EPOCH 73 ...
Validation Accuracy = 0.974

EPOCH 74 ...
Validation Accuracy = 0.980
INFO:tensorflow:./data/models/model_1/model_1 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 75 ...
Validation Accuracy = 0.974

EPOCH 76 ...
Validation Accuracy = 0.974

EPOCH 77 ...
Validation Accuracy = 0.972

EPOCH 78 ...
Validation Accuracy = 0.979

EPOCH 79 ...
Validation Accuracy = 0.974

EPOCH 80 ...
Validation Accuracy = 0.968

EPOCH 81 ...
Validation Accuracy = 0.973

EPOCH 82 ...
Validation Accuracy = 0.979

EPOCH 83 ...
Validation Accuracy = 0.975

EPOCH 84 ...
Validation Accuracy = 0.973

EPOCH 85 ...
Validation Accuracy = 0.973

EPOCH 86 ...
Validation Accuracy = 0.975

EPOCH 87 ...
Validation Accuracy = 0.977

EPOCH 88 ...
Validation Accuracy = 0.975

EPOCH 89 ...
Validation Accuracy = 0.976

EPOCH 90 ...
Validation Accuracy = 0.975

EPOCH 91 ...
Validation Accuracy = 0.978

EPOCH 92 ...
Validation Accuracy = 0.972

EPOCH 93 ...
Validation Accuracy = 0.973

EPOCH 94 ...
Validation Accuracy = 0.976

EPOCH 95 ...
Validation Accuracy = 0.973

EPOCH 96 ...
Validation Accuracy = 0.977

EPOCH 97 ...
Validation Accuracy = 0.968

EPOCH 98 ...
Validation Accuracy = 0.975

EPOCH 99 ...
Validation Accuracy = 0.979

EPOCH 100 ...
Validation Accuracy = 0.978

Best model - epoch: 74, best validation accuracy: 0.980
Test Accuracy = 0.962
###model_2
Training...

EPOCH 1 ...
Validation Accuracy = 0.124
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.472
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.713
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.840
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.899
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.911
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.932
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.949
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.947

EPOCH 10 ...
Validation Accuracy = 0.947

EPOCH 11 ...
Validation Accuracy = 0.954
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 12 ...
Validation Accuracy = 0.954
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 13 ...
Validation Accuracy = 0.949

EPOCH 14 ...
Validation Accuracy = 0.953

EPOCH 15 ...
Validation Accuracy = 0.966
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 16 ...
Validation Accuracy = 0.961

EPOCH 17 ...
Validation Accuracy = 0.969
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 18 ...
Validation Accuracy = 0.973
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 19 ...
Validation Accuracy = 0.964

EPOCH 20 ...
Validation Accuracy = 0.956

EPOCH 21 ...
Validation Accuracy = 0.952

EPOCH 22 ...
Validation Accuracy = 0.974
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 23 ...
Validation Accuracy = 0.966

EPOCH 24 ...
Validation Accuracy = 0.969

EPOCH 25 ...
Validation Accuracy = 0.972

EPOCH 26 ...
Validation Accuracy = 0.964

EPOCH 27 ...
Validation Accuracy = 0.977
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 28 ...
Validation Accuracy = 0.973

EPOCH 29 ...
Validation Accuracy = 0.971

EPOCH 30 ...
Validation Accuracy = 0.978
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 31 ...
Validation Accuracy = 0.971

EPOCH 32 ...
Validation Accuracy = 0.976

EPOCH 33 ...
Validation Accuracy = 0.978

EPOCH 34 ...
Validation Accuracy = 0.976

EPOCH 35 ...
Validation Accuracy = 0.969

EPOCH 36 ...
Validation Accuracy = 0.974

EPOCH 37 ...
Validation Accuracy = 0.974

EPOCH 38 ...
Validation Accuracy = 0.979
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 39 ...
Validation Accuracy = 0.983
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 40 ...
Validation Accuracy = 0.977

EPOCH 41 ...
Validation Accuracy = 0.981

EPOCH 42 ...
Validation Accuracy = 0.967

EPOCH 43 ...
Validation Accuracy = 0.976

EPOCH 44 ...
Validation Accuracy = 0.975

EPOCH 45 ...
Validation Accuracy = 0.980

EPOCH 46 ...
Validation Accuracy = 0.978

EPOCH 47 ...
Validation Accuracy = 0.981

EPOCH 48 ...
Validation Accuracy = 0.980

EPOCH 49 ...
Validation Accuracy = 0.971

EPOCH 50 ...
Validation Accuracy = 0.976

EPOCH 51 ...
Validation Accuracy = 0.976

EPOCH 52 ...
Validation Accuracy = 0.981

EPOCH 53 ...
Validation Accuracy = 0.968

EPOCH 54 ...
Validation Accuracy = 0.981

EPOCH 55 ...
Validation Accuracy = 0.978

EPOCH 56 ...
Validation Accuracy = 0.981

EPOCH 57 ...
Validation Accuracy = 0.977

EPOCH 58 ...
Validation Accuracy = 0.976

EPOCH 59 ...
Validation Accuracy = 0.979

EPOCH 60 ...
Validation Accuracy = 0.979

EPOCH 61 ...
Validation Accuracy = 0.976

EPOCH 62 ...
Validation Accuracy = 0.976

EPOCH 63 ...
Validation Accuracy = 0.979

EPOCH 64 ...
Validation Accuracy = 0.978

EPOCH 65 ...
Validation Accuracy = 0.978

EPOCH 66 ...
Validation Accuracy = 0.976

EPOCH 67 ...
Validation Accuracy = 0.982

EPOCH 68 ...
Validation Accuracy = 0.984
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 69 ...
Validation Accuracy = 0.983

EPOCH 70 ...
Validation Accuracy = 0.981

EPOCH 71 ...
Validation Accuracy = 0.980

EPOCH 72 ...
Validation Accuracy = 0.978

EPOCH 73 ...
Validation Accuracy = 0.978

EPOCH 74 ...
Validation Accuracy = 0.978

EPOCH 75 ...
Validation Accuracy = 0.980

EPOCH 76 ...
Validation Accuracy = 0.982

EPOCH 77 ...
Validation Accuracy = 0.976

EPOCH 78 ...
Validation Accuracy = 0.984
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 79 ...
Validation Accuracy = 0.982

EPOCH 80 ...
Validation Accuracy = 0.980

EPOCH 81 ...
Validation Accuracy = 0.978

EPOCH 82 ...
Validation Accuracy = 0.983

EPOCH 83 ...
Validation Accuracy = 0.975

EPOCH 84 ...
Validation Accuracy = 0.977

EPOCH 85 ...
Validation Accuracy = 0.977

EPOCH 86 ...
Validation Accuracy = 0.973

EPOCH 87 ...
Validation Accuracy = 0.976

EPOCH 88 ...
Validation Accuracy = 0.982

EPOCH 89 ...
Validation Accuracy = 0.980

EPOCH 90 ...
Validation Accuracy = 0.980

EPOCH 91 ...
Validation Accuracy = 0.975

EPOCH 92 ...
Validation Accuracy = 0.975

EPOCH 93 ...
Validation Accuracy = 0.982

EPOCH 94 ...
Validation Accuracy = 0.983

EPOCH 95 ...
Validation Accuracy = 0.976

EPOCH 96 ...
Validation Accuracy = 0.977

EPOCH 97 ...
Validation Accuracy = 0.984

EPOCH 98 ...
Validation Accuracy = 0.982

EPOCH 99 ...
Validation Accuracy = 0.984
INFO:tensorflow:./data/models/model_2/model_2 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 100 ...
Validation Accuracy = 0.978

Best model - epoch: 99, best validation accuracy: 0.984
Test Accuracy = 0.965
###model_3
Training...

EPOCH 1 ...
Validation Accuracy = 0.089
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.366
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.624
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.799
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.858
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.892
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.917
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.930
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.927

EPOCH 10 ...
Validation Accuracy = 0.943
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 11 ...
Validation Accuracy = 0.938

EPOCH 12 ...
Validation Accuracy = 0.937

EPOCH 13 ...
Validation Accuracy = 0.955
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 14 ...
Validation Accuracy = 0.944

EPOCH 15 ...
Validation Accuracy = 0.942

EPOCH 16 ...
Validation Accuracy = 0.958
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 17 ...
Validation Accuracy = 0.947

EPOCH 18 ...
Validation Accuracy = 0.955

EPOCH 19 ...
Validation Accuracy = 0.946

EPOCH 20 ...
Validation Accuracy = 0.955

EPOCH 21 ...
Validation Accuracy = 0.957

EPOCH 22 ...
Validation Accuracy = 0.954

EPOCH 23 ...
Validation Accuracy = 0.959
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 24 ...
Validation Accuracy = 0.955

EPOCH 25 ...
Validation Accuracy = 0.958

EPOCH 26 ...
Validation Accuracy = 0.957

EPOCH 27 ...
Validation Accuracy = 0.954

EPOCH 28 ...
Validation Accuracy = 0.953

EPOCH 29 ...
Validation Accuracy = 0.964
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 30 ...
Validation Accuracy = 0.958

EPOCH 31 ...
Validation Accuracy = 0.962

EPOCH 32 ...
Validation Accuracy = 0.953

EPOCH 33 ...
Validation Accuracy = 0.959

EPOCH 34 ...
Validation Accuracy = 0.966
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 35 ...
Validation Accuracy = 0.960

EPOCH 36 ...
Validation Accuracy = 0.963

EPOCH 37 ...
Validation Accuracy = 0.958

EPOCH 38 ...
Validation Accuracy = 0.970
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 39 ...
Validation Accuracy = 0.972
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 40 ...
Validation Accuracy = 0.962

EPOCH 41 ...
Validation Accuracy = 0.960

EPOCH 42 ...
Validation Accuracy = 0.961

EPOCH 43 ...
Validation Accuracy = 0.966

EPOCH 44 ...
Validation Accuracy = 0.966

EPOCH 45 ...
Validation Accuracy = 0.966

EPOCH 46 ...
Validation Accuracy = 0.963

EPOCH 47 ...
Validation Accuracy = 0.966

EPOCH 48 ...
Validation Accuracy = 0.963

EPOCH 49 ...
Validation Accuracy = 0.971

EPOCH 50 ...
Validation Accuracy = 0.968

EPOCH 51 ...
Validation Accuracy = 0.954

EPOCH 52 ...
Validation Accuracy = 0.964

EPOCH 53 ...
Validation Accuracy = 0.970

EPOCH 54 ...
Validation Accuracy = 0.973
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 55 ...
Validation Accuracy = 0.973

EPOCH 56 ...
Validation Accuracy = 0.960

EPOCH 57 ...
Validation Accuracy = 0.971

EPOCH 58 ...
Validation Accuracy = 0.975
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 59 ...
Validation Accuracy = 0.973

EPOCH 60 ...
Validation Accuracy = 0.979
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 61 ...
Validation Accuracy = 0.983
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 62 ...
Validation Accuracy = 0.968

EPOCH 63 ...
Validation Accuracy = 0.969

EPOCH 64 ...
Validation Accuracy = 0.967

EPOCH 65 ...
Validation Accuracy = 0.974

EPOCH 66 ...
Validation Accuracy = 0.979

EPOCH 67 ...
Validation Accuracy = 0.975

EPOCH 68 ...
Validation Accuracy = 0.974

EPOCH 69 ...
Validation Accuracy = 0.970

EPOCH 70 ...
Validation Accuracy = 0.979

EPOCH 71 ...
Validation Accuracy = 0.970

EPOCH 72 ...
Validation Accuracy = 0.977

EPOCH 73 ...
Validation Accuracy = 0.979

EPOCH 74 ...
Validation Accuracy = 0.972

EPOCH 75 ...
Validation Accuracy = 0.966

EPOCH 76 ...
Validation Accuracy = 0.975

EPOCH 77 ...
Validation Accuracy = 0.979

EPOCH 78 ...
Validation Accuracy = 0.977

EPOCH 79 ...
Validation Accuracy = 0.970

EPOCH 80 ...
Validation Accuracy = 0.972

EPOCH 81 ...
Validation Accuracy = 0.966

EPOCH 82 ...
Validation Accuracy = 0.967

EPOCH 83 ...
Validation Accuracy = 0.969

EPOCH 84 ...
Validation Accuracy = 0.973

EPOCH 85 ...
Validation Accuracy = 0.971

EPOCH 86 ...
Validation Accuracy = 0.980

EPOCH 87 ...
Validation Accuracy = 0.976

EPOCH 88 ...
Validation Accuracy = 0.980

EPOCH 89 ...
Validation Accuracy = 0.978

EPOCH 90 ...
Validation Accuracy = 0.983
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 91 ...
Validation Accuracy = 0.984
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 92 ...
Validation Accuracy = 0.979

EPOCH 93 ...
Validation Accuracy = 0.981

EPOCH 94 ...
Validation Accuracy = 0.978

EPOCH 95 ...
Validation Accuracy = 0.976

EPOCH 96 ...
Validation Accuracy = 0.983

EPOCH 97 ...
Validation Accuracy = 0.983

EPOCH 98 ...
Validation Accuracy = 0.985
INFO:tensorflow:./data/models/model_3/model_3 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 99 ...
Validation Accuracy = 0.980

EPOCH 100 ...
Validation Accuracy = 0.983

Best model - epoch: 98, best validation accuracy: 0.985
Test Accuracy = 0.966
###model_4
Training...

EPOCH 1 ...
Validation Accuracy = 0.091
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.398
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.640
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.809
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.843
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.912
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.917
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.937
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.936

EPOCH 10 ...
Validation Accuracy = 0.944
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 11 ...
Validation Accuracy = 0.952
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 12 ...
Validation Accuracy = 0.951

EPOCH 13 ...
Validation Accuracy = 0.939

EPOCH 14 ...
Validation Accuracy = 0.954
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 15 ...
Validation Accuracy = 0.956
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 16 ...
Validation Accuracy = 0.952

EPOCH 17 ...
Validation Accuracy = 0.961
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 18 ...
Validation Accuracy = 0.963
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 19 ...
Validation Accuracy = 0.960

EPOCH 20 ...
Validation Accuracy = 0.960

EPOCH 21 ...
Validation Accuracy = 0.960

EPOCH 22 ...
Validation Accuracy = 0.957

EPOCH 23 ...
Validation Accuracy = 0.966
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 24 ...
Validation Accuracy = 0.967
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 25 ...
Validation Accuracy = 0.961

EPOCH 26 ...
Validation Accuracy = 0.965

EPOCH 27 ...
Validation Accuracy = 0.971
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 28 ...
Validation Accuracy = 0.971

EPOCH 29 ...
Validation Accuracy = 0.966

EPOCH 30 ...
Validation Accuracy = 0.967

EPOCH 31 ...
Validation Accuracy = 0.969

EPOCH 32 ...
Validation Accuracy = 0.964

EPOCH 33 ...
Validation Accuracy = 0.970

EPOCH 34 ...
Validation Accuracy = 0.973
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 35 ...
Validation Accuracy = 0.968

EPOCH 36 ...
Validation Accuracy = 0.967

EPOCH 37 ...
Validation Accuracy = 0.966

EPOCH 38 ...
Validation Accuracy = 0.968

EPOCH 39 ...
Validation Accuracy = 0.973

EPOCH 40 ...
Validation Accuracy = 0.970

EPOCH 41 ...
Validation Accuracy = 0.974
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 42 ...
Validation Accuracy = 0.971

EPOCH 43 ...
Validation Accuracy = 0.980
INFO:tensorflow:./data/models/model_4/model_4 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 44 ...
Validation Accuracy = 0.971

EPOCH 45 ...
Validation Accuracy = 0.974

EPOCH 46 ...
Validation Accuracy = 0.972

EPOCH 47 ...
Validation Accuracy = 0.974

EPOCH 48 ...
Validation Accuracy = 0.976

EPOCH 49 ...
Validation Accuracy = 0.972

EPOCH 50 ...
Validation Accuracy = 0.971

EPOCH 51 ...
Validation Accuracy = 0.973

EPOCH 52 ...
Validation Accuracy = 0.966

EPOCH 53 ...
Validation Accuracy = 0.969

EPOCH 54 ...
Validation Accuracy = 0.962

EPOCH 55 ...
Validation Accuracy = 0.975

EPOCH 56 ...
Validation Accuracy = 0.977

EPOCH 57 ...
Validation Accuracy = 0.972

EPOCH 58 ...
Validation Accuracy = 0.974

EPOCH 59 ...
Validation Accuracy = 0.978

EPOCH 60 ...
Validation Accuracy = 0.977

EPOCH 61 ...
Validation Accuracy = 0.970

EPOCH 62 ...
Validation Accuracy = 0.978

EPOCH 63 ...
Validation Accuracy = 0.963

EPOCH 64 ...
Validation Accuracy = 0.971

EPOCH 65 ...
Validation Accuracy = 0.976

EPOCH 66 ...
Validation Accuracy = 0.968

EPOCH 67 ...
Validation Accuracy = 0.969

EPOCH 68 ...
Validation Accuracy = 0.973

EPOCH 69 ...
Validation Accuracy = 0.978

EPOCH 70 ...
Validation Accuracy = 0.968

EPOCH 71 ...
Validation Accuracy = 0.964

EPOCH 72 ...
Validation Accuracy = 0.971

EPOCH 73 ...
Validation Accuracy = 0.976

EPOCH 74 ...
Validation Accuracy = 0.970

EPOCH 75 ...
Validation Accuracy = 0.978

EPOCH 76 ...
Validation Accuracy = 0.972

EPOCH 77 ...
Validation Accuracy = 0.975

EPOCH 78 ...
Validation Accuracy = 0.976

EPOCH 79 ...
Validation Accuracy = 0.973

EPOCH 80 ...
Validation Accuracy = 0.975

EPOCH 81 ...
Validation Accuracy = 0.975

EPOCH 82 ...
Validation Accuracy = 0.973

EPOCH 83 ...
Validation Accuracy = 0.977

EPOCH 84 ...
Validation Accuracy = 0.975

EPOCH 85 ...
Validation Accuracy = 0.970

EPOCH 86 ...
Validation Accuracy = 0.967

EPOCH 87 ...
Validation Accuracy = 0.970

EPOCH 88 ...
Validation Accuracy = 0.964

EPOCH 89 ...
Validation Accuracy = 0.971

EPOCH 90 ...
Validation Accuracy = 0.975

EPOCH 91 ...
Validation Accuracy = 0.974

EPOCH 92 ...
Validation Accuracy = 0.972

EPOCH 93 ...
Validation Accuracy = 0.977

EPOCH 94 ...
Validation Accuracy = 0.976

EPOCH 95 ...
Validation Accuracy = 0.976

EPOCH 96 ...
Validation Accuracy = 0.975

EPOCH 97 ...
Validation Accuracy = 0.975

EPOCH 98 ...
Validation Accuracy = 0.973

EPOCH 99 ...
Validation Accuracy = 0.975

EPOCH 100 ...
Validation Accuracy = 0.975

Best model - epoch: 43, best validation accuracy: 0.980
Test Accuracy = 0.957
###model_5
Training...

EPOCH 1 ...
Validation Accuracy = 0.105
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.434
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.718
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.837
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.901
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.930
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.918

EPOCH 8 ...
Validation Accuracy = 0.941
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.936

EPOCH 10 ...
Validation Accuracy = 0.955
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 11 ...
Validation Accuracy = 0.947

EPOCH 12 ...
Validation Accuracy = 0.949

EPOCH 13 ...
Validation Accuracy = 0.943

EPOCH 14 ...
Validation Accuracy = 0.946

EPOCH 15 ...
Validation Accuracy = 0.946

EPOCH 16 ...
Validation Accuracy = 0.952

EPOCH 17 ...
Validation Accuracy = 0.952

EPOCH 18 ...
Validation Accuracy = 0.946

EPOCH 19 ...
Validation Accuracy = 0.955

EPOCH 20 ...
Validation Accuracy = 0.950

EPOCH 21 ...
Validation Accuracy = 0.952

EPOCH 22 ...
Validation Accuracy = 0.953

EPOCH 23 ...
Validation Accuracy = 0.959
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 24 ...
Validation Accuracy = 0.957

EPOCH 25 ...
Validation Accuracy = 0.956

EPOCH 26 ...
Validation Accuracy = 0.961
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 27 ...
Validation Accuracy = 0.966
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 28 ...
Validation Accuracy = 0.962

EPOCH 29 ...
Validation Accuracy = 0.960

EPOCH 30 ...
Validation Accuracy = 0.960

EPOCH 31 ...
Validation Accuracy = 0.962

EPOCH 32 ...
Validation Accuracy = 0.961

EPOCH 33 ...
Validation Accuracy = 0.962

EPOCH 34 ...
Validation Accuracy = 0.955

EPOCH 35 ...
Validation Accuracy = 0.966

EPOCH 36 ...
Validation Accuracy = 0.961

EPOCH 37 ...
Validation Accuracy = 0.953

EPOCH 38 ...
Validation Accuracy = 0.961

EPOCH 39 ...
Validation Accuracy = 0.960

EPOCH 40 ...
Validation Accuracy = 0.965

EPOCH 41 ...
Validation Accuracy = 0.955

EPOCH 42 ...
Validation Accuracy = 0.964

EPOCH 43 ...
Validation Accuracy = 0.958

EPOCH 44 ...
Validation Accuracy = 0.963

EPOCH 45 ...
Validation Accuracy = 0.966

EPOCH 46 ...
Validation Accuracy = 0.963

EPOCH 47 ...
Validation Accuracy = 0.968
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 48 ...
Validation Accuracy = 0.968
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 49 ...
Validation Accuracy = 0.965

EPOCH 50 ...
Validation Accuracy = 0.961

EPOCH 51 ...
Validation Accuracy = 0.966

EPOCH 52 ...
Validation Accuracy = 0.963

EPOCH 53 ...
Validation Accuracy = 0.966

EPOCH 54 ...
Validation Accuracy = 0.973
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 55 ...
Validation Accuracy = 0.972

EPOCH 56 ...
Validation Accuracy = 0.960

EPOCH 57 ...
Validation Accuracy = 0.971

EPOCH 58 ...
Validation Accuracy = 0.961

EPOCH 59 ...
Validation Accuracy = 0.973
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 60 ...
Validation Accuracy = 0.969

EPOCH 61 ...
Validation Accuracy = 0.972

EPOCH 62 ...
Validation Accuracy = 0.971

EPOCH 63 ...
Validation Accuracy = 0.969

EPOCH 64 ...
Validation Accuracy = 0.964

EPOCH 65 ...
Validation Accuracy = 0.968

EPOCH 66 ...
Validation Accuracy = 0.969

EPOCH 67 ...
Validation Accuracy = 0.975
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 68 ...
Validation Accuracy = 0.968

EPOCH 69 ...
Validation Accuracy = 0.976
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 70 ...
Validation Accuracy = 0.973

EPOCH 71 ...
Validation Accuracy = 0.970

EPOCH 72 ...
Validation Accuracy = 0.974

EPOCH 73 ...
Validation Accuracy = 0.970

EPOCH 74 ...
Validation Accuracy = 0.975

EPOCH 75 ...
Validation Accuracy = 0.975

EPOCH 76 ...
Validation Accuracy = 0.967

EPOCH 77 ...
Validation Accuracy = 0.971

EPOCH 78 ...
Validation Accuracy = 0.976
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 79 ...
Validation Accuracy = 0.968

EPOCH 80 ...
Validation Accuracy = 0.976

EPOCH 81 ...
Validation Accuracy = 0.965

EPOCH 82 ...
Validation Accuracy = 0.974

EPOCH 83 ...
Validation Accuracy = 0.965

EPOCH 84 ...
Validation Accuracy = 0.975

EPOCH 85 ...
Validation Accuracy = 0.972

EPOCH 86 ...
Validation Accuracy = 0.972

EPOCH 87 ...
Validation Accuracy = 0.979
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 88 ...
Validation Accuracy = 0.966

EPOCH 89 ...
Validation Accuracy = 0.969

EPOCH 90 ...
Validation Accuracy = 0.968

EPOCH 91 ...
Validation Accuracy = 0.977

EPOCH 92 ...
Validation Accuracy = 0.975

EPOCH 93 ...
Validation Accuracy = 0.982
INFO:tensorflow:./data/models/model_5/model_5 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 94 ...
Validation Accuracy = 0.968

EPOCH 95 ...
Validation Accuracy = 0.978

EPOCH 96 ...
Validation Accuracy = 0.970

EPOCH 97 ...
Validation Accuracy = 0.978

EPOCH 98 ...
Validation Accuracy = 0.978

EPOCH 99 ...
Validation Accuracy = 0.973

EPOCH 100 ...
Validation Accuracy = 0.978

Best model - epoch: 93, best validation accuracy: 0.982
Test Accuracy = 0.962
###model_6
Training...

EPOCH 1 ...
Validation Accuracy = 0.094
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.382
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.670
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.820
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.885
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.907
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.928
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.931
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.937
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 10 ...
Validation Accuracy = 0.945
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 11 ...
Validation Accuracy = 0.934

EPOCH 12 ...
Validation Accuracy = 0.907

EPOCH 13 ...
Validation Accuracy = 0.941

EPOCH 14 ...
Validation Accuracy = 0.944

EPOCH 15 ...
Validation Accuracy = 0.955
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 16 ...
Validation Accuracy = 0.944

EPOCH 17 ...
Validation Accuracy = 0.951

EPOCH 18 ...
Validation Accuracy = 0.959
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 19 ...
Validation Accuracy = 0.949

EPOCH 20 ...
Validation Accuracy = 0.946

EPOCH 21 ...
Validation Accuracy = 0.948

EPOCH 22 ...
Validation Accuracy = 0.956

EPOCH 23 ...
Validation Accuracy = 0.954

EPOCH 24 ...
Validation Accuracy = 0.951

EPOCH 25 ...
Validation Accuracy = 0.951

EPOCH 26 ...
Validation Accuracy = 0.962
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 27 ...
Validation Accuracy = 0.948

EPOCH 28 ...
Validation Accuracy = 0.959

EPOCH 29 ...
Validation Accuracy = 0.953

EPOCH 30 ...
Validation Accuracy = 0.955

EPOCH 31 ...
Validation Accuracy = 0.955

EPOCH 32 ...
Validation Accuracy = 0.955

EPOCH 33 ...
Validation Accuracy = 0.950

EPOCH 34 ...
Validation Accuracy = 0.952

EPOCH 35 ...
Validation Accuracy = 0.961

EPOCH 36 ...
Validation Accuracy = 0.960

EPOCH 37 ...
Validation Accuracy = 0.964
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 38 ...
Validation Accuracy = 0.960

EPOCH 39 ...
Validation Accuracy = 0.964

EPOCH 40 ...
Validation Accuracy = 0.961

EPOCH 41 ...
Validation Accuracy = 0.957

EPOCH 42 ...
Validation Accuracy = 0.967
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 43 ...
Validation Accuracy = 0.959

EPOCH 44 ...
Validation Accuracy = 0.968
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 45 ...
Validation Accuracy = 0.958

EPOCH 46 ...
Validation Accuracy = 0.960

EPOCH 47 ...
Validation Accuracy = 0.966

EPOCH 48 ...
Validation Accuracy = 0.970
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 49 ...
Validation Accuracy = 0.974
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 50 ...
Validation Accuracy = 0.962

EPOCH 51 ...
Validation Accuracy = 0.968

EPOCH 52 ...
Validation Accuracy = 0.966

EPOCH 53 ...
Validation Accuracy = 0.966

EPOCH 54 ...
Validation Accuracy = 0.966

EPOCH 55 ...
Validation Accuracy = 0.961

EPOCH 56 ...
Validation Accuracy = 0.971

EPOCH 57 ...
Validation Accuracy = 0.961

EPOCH 58 ...
Validation Accuracy = 0.973

EPOCH 59 ...
Validation Accuracy = 0.964

EPOCH 60 ...
Validation Accuracy = 0.966

EPOCH 61 ...
Validation Accuracy = 0.966

EPOCH 62 ...
Validation Accuracy = 0.967

EPOCH 63 ...
Validation Accuracy = 0.966

EPOCH 64 ...
Validation Accuracy = 0.966

EPOCH 65 ...
Validation Accuracy = 0.966

EPOCH 66 ...
Validation Accuracy = 0.970

EPOCH 67 ...
Validation Accuracy = 0.972

EPOCH 68 ...
Validation Accuracy = 0.971

EPOCH 69 ...
Validation Accuracy = 0.968

EPOCH 70 ...
Validation Accuracy = 0.971

EPOCH 71 ...
Validation Accuracy = 0.969

EPOCH 72 ...
Validation Accuracy = 0.975
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 73 ...
Validation Accuracy = 0.973

EPOCH 74 ...
Validation Accuracy = 0.965

EPOCH 75 ...
Validation Accuracy = 0.977
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 76 ...
Validation Accuracy = 0.967

EPOCH 77 ...
Validation Accuracy = 0.977

EPOCH 78 ...
Validation Accuracy = 0.967

EPOCH 79 ...
Validation Accuracy = 0.970

EPOCH 80 ...
Validation Accuracy = 0.972

EPOCH 81 ...
Validation Accuracy = 0.976

EPOCH 82 ...
Validation Accuracy = 0.973

EPOCH 83 ...
Validation Accuracy = 0.972

EPOCH 84 ...
Validation Accuracy = 0.975

EPOCH 85 ...
Validation Accuracy = 0.974

EPOCH 86 ...
Validation Accuracy = 0.965

EPOCH 87 ...
Validation Accuracy = 0.971

EPOCH 88 ...
Validation Accuracy = 0.981
INFO:tensorflow:./data/models/model_6/model_6 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 89 ...
Validation Accuracy = 0.977

EPOCH 90 ...
Validation Accuracy = 0.976

EPOCH 91 ...
Validation Accuracy = 0.975

EPOCH 92 ...
Validation Accuracy = 0.971

EPOCH 93 ...
Validation Accuracy = 0.971

EPOCH 94 ...
Validation Accuracy = 0.976

EPOCH 95 ...
Validation Accuracy = 0.971

EPOCH 96 ...
Validation Accuracy = 0.972

EPOCH 97 ...
Validation Accuracy = 0.978

EPOCH 98 ...
Validation Accuracy = 0.974

EPOCH 99 ...
Validation Accuracy = 0.969

EPOCH 100 ...
Validation Accuracy = 0.973

Best model - epoch: 88, best validation accuracy: 0.981
Test Accuracy = 0.965
###model_7
Training...

EPOCH 1 ...
Validation Accuracy = 0.102
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.522
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.722
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.843
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.883
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.896
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.917
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.933
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.934
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 10 ...
Validation Accuracy = 0.931

EPOCH 11 ...
Validation Accuracy = 0.949
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 12 ...
Validation Accuracy = 0.949

EPOCH 13 ...
Validation Accuracy = 0.951
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 14 ...
Validation Accuracy = 0.948

EPOCH 15 ...
Validation Accuracy = 0.960
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 16 ...
Validation Accuracy = 0.957

EPOCH 17 ...
Validation Accuracy = 0.957

EPOCH 18 ...
Validation Accuracy = 0.950

EPOCH 19 ...
Validation Accuracy = 0.955

EPOCH 20 ...
Validation Accuracy = 0.963
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 21 ...
Validation Accuracy = 0.960

EPOCH 22 ...
Validation Accuracy = 0.958

EPOCH 23 ...
Validation Accuracy = 0.957

EPOCH 24 ...
Validation Accuracy = 0.965
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 25 ...
Validation Accuracy = 0.958

EPOCH 26 ...
Validation Accuracy = 0.961

EPOCH 27 ...
Validation Accuracy = 0.967
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 28 ...
Validation Accuracy = 0.962

EPOCH 29 ...
Validation Accuracy = 0.957

EPOCH 30 ...
Validation Accuracy = 0.958

EPOCH 31 ...
Validation Accuracy = 0.967

EPOCH 32 ...
Validation Accuracy = 0.970
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 33 ...
Validation Accuracy = 0.964

EPOCH 34 ...
Validation Accuracy = 0.967

EPOCH 35 ...
Validation Accuracy = 0.962

EPOCH 36 ...
Validation Accuracy = 0.962

EPOCH 37 ...
Validation Accuracy = 0.962

EPOCH 38 ...
Validation Accuracy = 0.972
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 39 ...
Validation Accuracy = 0.965

EPOCH 40 ...
Validation Accuracy = 0.966

EPOCH 41 ...
Validation Accuracy = 0.967

EPOCH 42 ...
Validation Accuracy = 0.968

EPOCH 43 ...
Validation Accuracy = 0.957

EPOCH 44 ...
Validation Accuracy = 0.971

EPOCH 45 ...
Validation Accuracy = 0.965

EPOCH 46 ...
Validation Accuracy = 0.974
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 47 ...
Validation Accuracy = 0.971

EPOCH 48 ...
Validation Accuracy = 0.969

EPOCH 49 ...
Validation Accuracy = 0.966

EPOCH 50 ...
Validation Accuracy = 0.964

EPOCH 51 ...
Validation Accuracy = 0.962

EPOCH 52 ...
Validation Accuracy = 0.969

EPOCH 53 ...
Validation Accuracy = 0.970

EPOCH 54 ...
Validation Accuracy = 0.966

EPOCH 55 ...
Validation Accuracy = 0.967

EPOCH 56 ...
Validation Accuracy = 0.972

EPOCH 57 ...
Validation Accuracy = 0.973

EPOCH 58 ...
Validation Accuracy = 0.971

EPOCH 59 ...
Validation Accuracy = 0.969

EPOCH 60 ...
Validation Accuracy = 0.977
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 61 ...
Validation Accuracy = 0.971

EPOCH 62 ...
Validation Accuracy = 0.971

EPOCH 63 ...
Validation Accuracy = 0.974

EPOCH 64 ...
Validation Accuracy = 0.969

EPOCH 65 ...
Validation Accuracy = 0.973

EPOCH 66 ...
Validation Accuracy = 0.971

EPOCH 67 ...
Validation Accuracy = 0.972

EPOCH 68 ...
Validation Accuracy = 0.970

EPOCH 69 ...
Validation Accuracy = 0.969

EPOCH 70 ...
Validation Accuracy = 0.974

EPOCH 71 ...
Validation Accuracy = 0.965

EPOCH 72 ...
Validation Accuracy = 0.978
INFO:tensorflow:./data/models/model_7/model_7 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 73 ...
Validation Accuracy = 0.972

EPOCH 74 ...
Validation Accuracy = 0.973

EPOCH 75 ...
Validation Accuracy = 0.968

EPOCH 76 ...
Validation Accuracy = 0.975

EPOCH 77 ...
Validation Accuracy = 0.974

EPOCH 78 ...
Validation Accuracy = 0.977

EPOCH 79 ...
Validation Accuracy = 0.972

EPOCH 80 ...
Validation Accuracy = 0.977

EPOCH 81 ...
Validation Accuracy = 0.973

EPOCH 82 ...
Validation Accuracy = 0.976

EPOCH 83 ...
Validation Accuracy = 0.970

EPOCH 84 ...
Validation Accuracy = 0.969

EPOCH 85 ...
Validation Accuracy = 0.965

EPOCH 86 ...
Validation Accuracy = 0.966

EPOCH 87 ...
Validation Accuracy = 0.973

EPOCH 88 ...
Validation Accuracy = 0.975

EPOCH 89 ...
Validation Accuracy = 0.973

EPOCH 90 ...
Validation Accuracy = 0.972

EPOCH 91 ...
Validation Accuracy = 0.974

EPOCH 92 ...
Validation Accuracy = 0.974

EPOCH 93 ...
Validation Accuracy = 0.976

EPOCH 94 ...
Validation Accuracy = 0.974

EPOCH 95 ...
Validation Accuracy = 0.974

EPOCH 96 ...
Validation Accuracy = 0.965

EPOCH 97 ...
Validation Accuracy = 0.973

EPOCH 98 ...
Validation Accuracy = 0.975

EPOCH 99 ...
Validation Accuracy = 0.974

EPOCH 100 ...
Validation Accuracy = 0.974

Best model - epoch: 72, best validation accuracy: 0.978
Test Accuracy = 0.965
###model_8
Training...

EPOCH 1 ...
Validation Accuracy = 0.101
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.336
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.557
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.753
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.851
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.882
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.916
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.930
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.928

EPOCH 10 ...
Validation Accuracy = 0.937
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 11 ...
Validation Accuracy = 0.943
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 12 ...
Validation Accuracy = 0.927

EPOCH 13 ...
Validation Accuracy = 0.939

EPOCH 14 ...
Validation Accuracy = 0.947
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 15 ...
Validation Accuracy = 0.948
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 16 ...
Validation Accuracy = 0.948
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 17 ...
Validation Accuracy = 0.955
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 18 ...
Validation Accuracy = 0.939

EPOCH 19 ...
Validation Accuracy = 0.950

EPOCH 20 ...
Validation Accuracy = 0.957
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 21 ...
Validation Accuracy = 0.952

EPOCH 22 ...
Validation Accuracy = 0.957
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 23 ...
Validation Accuracy = 0.953

EPOCH 24 ...
Validation Accuracy = 0.952

EPOCH 25 ...
Validation Accuracy = 0.950

EPOCH 26 ...
Validation Accuracy = 0.953

EPOCH 27 ...
Validation Accuracy = 0.951

EPOCH 28 ...
Validation Accuracy = 0.956

EPOCH 29 ...
Validation Accuracy = 0.960
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 30 ...
Validation Accuracy = 0.958

EPOCH 31 ...
Validation Accuracy = 0.959

EPOCH 32 ...
Validation Accuracy = 0.954

EPOCH 33 ...
Validation Accuracy = 0.944

EPOCH 34 ...
Validation Accuracy = 0.958

EPOCH 35 ...
Validation Accuracy = 0.964
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 36 ...
Validation Accuracy = 0.957

EPOCH 37 ...
Validation Accuracy = 0.964
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 38 ...
Validation Accuracy = 0.962

EPOCH 39 ...
Validation Accuracy = 0.962

EPOCH 40 ...
Validation Accuracy = 0.961

EPOCH 41 ...
Validation Accuracy = 0.961

EPOCH 42 ...
Validation Accuracy = 0.962

EPOCH 43 ...
Validation Accuracy = 0.961

EPOCH 44 ...
Validation Accuracy = 0.965
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 45 ...
Validation Accuracy = 0.961

EPOCH 46 ...
Validation Accuracy = 0.964

EPOCH 47 ...
Validation Accuracy = 0.956

EPOCH 48 ...
Validation Accuracy = 0.962

EPOCH 49 ...
Validation Accuracy = 0.966
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 50 ...
Validation Accuracy = 0.967
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 51 ...
Validation Accuracy = 0.965

EPOCH 52 ...
Validation Accuracy = 0.962

EPOCH 53 ...
Validation Accuracy = 0.965

EPOCH 54 ...
Validation Accuracy = 0.960

EPOCH 55 ...
Validation Accuracy = 0.962

EPOCH 56 ...
Validation Accuracy = 0.957

EPOCH 57 ...
Validation Accuracy = 0.960

EPOCH 58 ...
Validation Accuracy = 0.969
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 59 ...
Validation Accuracy = 0.967

EPOCH 60 ...
Validation Accuracy = 0.963

EPOCH 61 ...
Validation Accuracy = 0.963

EPOCH 62 ...
Validation Accuracy = 0.964

EPOCH 63 ...
Validation Accuracy = 0.966

EPOCH 64 ...
Validation Accuracy = 0.964

EPOCH 65 ...
Validation Accuracy = 0.960

EPOCH 66 ...
Validation Accuracy = 0.961

EPOCH 67 ...
Validation Accuracy = 0.962

EPOCH 68 ...
Validation Accuracy = 0.969

EPOCH 69 ...
Validation Accuracy = 0.964

EPOCH 70 ...
Validation Accuracy = 0.968

EPOCH 71 ...
Validation Accuracy = 0.969

EPOCH 72 ...
Validation Accuracy = 0.964

EPOCH 73 ...
Validation Accuracy = 0.965

EPOCH 74 ...
Validation Accuracy = 0.968

EPOCH 75 ...
Validation Accuracy = 0.967

EPOCH 76 ...
Validation Accuracy = 0.966

EPOCH 77 ...
Validation Accuracy = 0.963

EPOCH 78 ...
Validation Accuracy = 0.968

EPOCH 79 ...
Validation Accuracy = 0.963

EPOCH 80 ...
Validation Accuracy = 0.966

EPOCH 81 ...
Validation Accuracy = 0.971
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 82 ...
Validation Accuracy = 0.973
INFO:tensorflow:./data/models/model_8/model_8 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 83 ...
Validation Accuracy = 0.967

EPOCH 84 ...
Validation Accuracy = 0.971

EPOCH 85 ...
Validation Accuracy = 0.966

EPOCH 86 ...
Validation Accuracy = 0.968

EPOCH 87 ...
Validation Accuracy = 0.964

EPOCH 88 ...
Validation Accuracy = 0.968

EPOCH 89 ...
Validation Accuracy = 0.962

EPOCH 90 ...
Validation Accuracy = 0.964

EPOCH 91 ...
Validation Accuracy = 0.963

EPOCH 92 ...
Validation Accuracy = 0.968

EPOCH 93 ...
Validation Accuracy = 0.968

EPOCH 94 ...
Validation Accuracy = 0.970

EPOCH 95 ...
Validation Accuracy = 0.968

EPOCH 96 ...
Validation Accuracy = 0.965

EPOCH 97 ...
Validation Accuracy = 0.971

EPOCH 98 ...
Validation Accuracy = 0.965

EPOCH 99 ...
Validation Accuracy = 0.969

EPOCH 100 ...
Validation Accuracy = 0.963

Best model - epoch: 82, best validation accuracy: 0.973
Test Accuracy = 0.963
###model_9
Training...

EPOCH 1 ...
Validation Accuracy = 0.105
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 2 ...
Validation Accuracy = 0.464
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 3 ...
Validation Accuracy = 0.681
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 4 ...
Validation Accuracy = 0.818
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 5 ...
Validation Accuracy = 0.876
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 6 ...
Validation Accuracy = 0.901
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 7 ...
Validation Accuracy = 0.930
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 8 ...
Validation Accuracy = 0.931
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 9 ...
Validation Accuracy = 0.942
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 10 ...
Validation Accuracy = 0.946
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 11 ...
Validation Accuracy = 0.946
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 12 ...
Validation Accuracy = 0.947
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 13 ...
Validation Accuracy = 0.952
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 14 ...
Validation Accuracy = 0.954
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 15 ...
Validation Accuracy = 0.956
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 16 ...
Validation Accuracy = 0.956

EPOCH 17 ...
Validation Accuracy = 0.963
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 18 ...
Validation Accuracy = 0.958

EPOCH 19 ...
Validation Accuracy = 0.961

EPOCH 20 ...
Validation Accuracy = 0.949

EPOCH 21 ...
Validation Accuracy = 0.953

EPOCH 22 ...
Validation Accuracy = 0.963
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 23 ...
Validation Accuracy = 0.966
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 24 ...
Validation Accuracy = 0.953

EPOCH 25 ...
Validation Accuracy = 0.957

EPOCH 26 ...
Validation Accuracy = 0.966

EPOCH 27 ...
Validation Accuracy = 0.963

EPOCH 28 ...
Validation Accuracy = 0.960

EPOCH 29 ...
Validation Accuracy = 0.964

EPOCH 30 ...
Validation Accuracy = 0.960

EPOCH 31 ...
Validation Accuracy = 0.963

EPOCH 32 ...
Validation Accuracy = 0.958

EPOCH 33 ...
Validation Accuracy = 0.965

EPOCH 34 ...
Validation Accuracy = 0.965

EPOCH 35 ...
Validation Accuracy = 0.964

EPOCH 36 ...
Validation Accuracy = 0.968
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 37 ...
Validation Accuracy = 0.969
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 38 ...
Validation Accuracy = 0.971
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 39 ...
Validation Accuracy = 0.968

EPOCH 40 ...
Validation Accuracy = 0.968

EPOCH 41 ...
Validation Accuracy = 0.962

EPOCH 42 ...
Validation Accuracy = 0.975
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 43 ...
Validation Accuracy = 0.974

EPOCH 44 ...
Validation Accuracy = 0.975

EPOCH 45 ...
Validation Accuracy = 0.966

EPOCH 46 ...
Validation Accuracy = 0.969

EPOCH 47 ...
Validation Accuracy = 0.975

EPOCH 48 ...
Validation Accuracy = 0.968

EPOCH 49 ...
Validation Accuracy = 0.971

EPOCH 50 ...
Validation Accuracy = 0.974

EPOCH 51 ...
Validation Accuracy = 0.973

EPOCH 52 ...
Validation Accuracy = 0.969

EPOCH 53 ...
Validation Accuracy = 0.964

EPOCH 54 ...
Validation Accuracy = 0.972

EPOCH 55 ...
Validation Accuracy = 0.972

EPOCH 56 ...
Validation Accuracy = 0.976
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 57 ...
Validation Accuracy = 0.972

EPOCH 58 ...
Validation Accuracy = 0.970

EPOCH 59 ...
Validation Accuracy = 0.974

EPOCH 60 ...
Validation Accuracy = 0.974

EPOCH 61 ...
Validation Accuracy = 0.979
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 62 ...
Validation Accuracy = 0.976

EPOCH 63 ...
Validation Accuracy = 0.972

EPOCH 64 ...
Validation Accuracy = 0.969

EPOCH 65 ...
Validation Accuracy = 0.975

EPOCH 66 ...
Validation Accuracy = 0.968

EPOCH 67 ...
Validation Accuracy = 0.974

EPOCH 68 ...
Validation Accuracy = 0.974

EPOCH 69 ...
Validation Accuracy = 0.978

EPOCH 70 ...
Validation Accuracy = 0.973

EPOCH 71 ...
Validation Accuracy = 0.970

EPOCH 72 ...
Validation Accuracy = 0.973

EPOCH 73 ...
Validation Accuracy = 0.973

EPOCH 74 ...
Validation Accuracy = 0.978

EPOCH 75 ...
Validation Accuracy = 0.973

EPOCH 76 ...
Validation Accuracy = 0.968

EPOCH 77 ...
Validation Accuracy = 0.971

EPOCH 78 ...
Validation Accuracy = 0.976

EPOCH 79 ...
Validation Accuracy = 0.974

EPOCH 80 ...
Validation Accuracy = 0.980
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 81 ...
Validation Accuracy = 0.980
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 82 ...
Validation Accuracy = 0.978

EPOCH 83 ...
Validation Accuracy = 0.963

EPOCH 84 ...
Validation Accuracy = 0.973

EPOCH 85 ...
Validation Accuracy = 0.974

EPOCH 86 ...
Validation Accuracy = 0.975

EPOCH 87 ...
Validation Accuracy = 0.979

EPOCH 88 ...
Validation Accuracy = 0.975

EPOCH 89 ...
Validation Accuracy = 0.974

EPOCH 90 ...
Validation Accuracy = 0.974

EPOCH 91 ...
Validation Accuracy = 0.969

EPOCH 92 ...
Validation Accuracy = 0.977

EPOCH 93 ...
Validation Accuracy = 0.981
INFO:tensorflow:./data/models/model_9/model_9 is not in all_model_checkpoint_paths. Manually adding it.
Model saved

EPOCH 94 ...
Validation Accuracy = 0.978

EPOCH 95 ...
Validation Accuracy = 0.974

EPOCH 96 ...
Validation Accuracy = 0.976

EPOCH 97 ...
Validation Accuracy = 0.975

EPOCH 98 ...
Validation Accuracy = 0.980

EPOCH 99 ...
Validation Accuracy = 0.976

EPOCH 100 ...
Validation Accuracy = 0.978

Best model - epoch: 93, best validation accuracy: 0.981
Test Accuracy = 0.968

Individual Accuracies

In [73]:
def get_individual_accuracies(y, y_hat_arr):
    accuracy_arr = []
    image_cnt = y.shape[0]

    for j in tqdm(range(len(y_hat_arr))):
        correct_cnt = 0

        curr_y_hat = y_hat_arr[j]
        assert curr_y_hat.shape[0] == image_cnt, "%d, %d" % (curr_y_hat.shape[0], image_cnt)
        
        for i in range(image_cnt):
            if y[i] == curr_y_hat[i]:
                correct_cnt += 1
        
        accuracy = float(correct_cnt)/image_cnt
        accuracy_arr.append(accuracy)
        
    return accuracy_arr
Train Accuracy
In [74]:
def display_ensemble_accuracy(ensemble_accuracy_arr):
    print("Model\tAccuracy")
    print("================")
    for i, ensemble_accuracy in enumerate(ensemble_accuracy_arr):
        print("%d\t %.3f" % (i, ensemble_accuracy))    
In [75]:
n = 10
In [76]:
y_ensemble_train_hat_arr = predict_ensemble_models(X_train_preprocessed, 'model', 
                                                   n, batch_size=BATCH_SIZE)
ensemble_train_accuracy_arr = get_individual_accuracies(y_train_preprocessed, 
                                                        y_ensemble_train_hat_arr)

display_ensemble_accuracy(ensemble_train_accuracy_arr)
100%|██████████| 10/10 [00:29<00:00,  2.98s/it]
100%|██████████| 10/10 [00:00<00:00, 11.63it/s]
Model	Accuracy
================
0	 0.999
1	 1.000
2	 1.000
3	 1.000
4	 0.999
5	 0.999
6	 1.000
7	 0.999
8	 0.999
9	 1.000

Validation Accuracy
In [77]:
y_ensemble_valid_hat_arr = predict_ensemble_models(X_valid_preprocessed, 'model', 
                                                   n, batch_size=BATCH_SIZE)

ensemble_valid_accuracy_arr = get_individual_accuracies(y_valid_preprocessed, 
                                                        y_ensemble_valid_hat_arr)

display_ensemble_accuracy(ensemble_valid_accuracy_arr)
100%|██████████| 10/10 [00:09<00:00,  1.03s/it]
100%|██████████| 10/10 [00:00<00:00, 88.06it/s]
Model	Accuracy
================
0	 0.979
1	 0.980
2	 0.984
3	 0.985
4	 0.980
5	 0.982
6	 0.981
7	 0.978
8	 0.973
9	 0.981

Test Accuracy
In [78]:
y_ensemble_test_hat_arr = predict_ensemble_models(X_test_preprocessed, 'model', 
                                                   n, batch_size=BATCH_SIZE)
ensemble_test_accuracy_arr = get_individual_accuracies(y_test_preprocessed, y_ensemble_test_hat_arr)

display_ensemble_accuracy(ensemble_test_accuracy_arr)
100%|██████████| 10/10 [00:17<00:00,  1.79s/it]
100%|██████████| 10/10 [00:00<00:00, 30.45it/s]
Model	Accuracy
================
0	 0.962
1	 0.962
2	 0.965
3	 0.966
4	 0.957
5	 0.962
6	 0.965
7	 0.965
8	 0.963
9	 0.968

In [79]:
# data to plot
n_groups = 10

# create plot
fig, ax = plt.subplots(figsize=(20, 10))
index = np.arange(n_groups)
bar_width = 0.25
opacity = 0.8
 
rects1 = plt.bar(index, ensemble_train_accuracy_arr, bar_width,
                 alpha=opacity,
                 color='#B3B3B3',
                 label='Train')
 
rects2 = plt.bar(index + bar_width, ensemble_valid_accuracy_arr, bar_width,
                 alpha=opacity,
                 color='#1F1A4F',
                 label='Validation')

rects3 = plt.bar(index + 2*bar_width, ensemble_test_accuracy_arr, bar_width,
                 alpha=opacity,
                 color='#C82027',
                 label='Test')
 
plt.xlabel('Models').set_fontsize(24)
plt.ylabel('Accuracies').set_fontsize(24)
plt.title('Individual Model Accuracies').set_fontsize(32)
plt.xticks(index + bar_width, range(n_groups))
plt.legend(fontsize='x-large')
 
plt.tight_layout()
plt.show()

Ensembled Accuracies

In [80]:
def predict_model_softmax_logits(X, model_name, batch_size):
    if X.shape[0] < batch_size:
        batch_size = X.shape[0]
        
    model_dir = '%s/models/%s' % (data_dir, model_name)
    saver = tf.train.Saver()
    
    softmax_logit_vals_arr = []
    
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())  
        saver.restore(sess, tf.train.latest_checkpoint(model_dir))
        
        softmax_logits = tf.nn.softmax(logits)
        
        start_idx = 0
        end_idx = 0
        while start_idx < X.shape[0]:
            end_idx += batch_size
            end_idx = min(end_idx, X.shape[0])
            softmax_logit_vals = sess.run(softmax_logits, feed_dict={x: X[start_idx:end_idx], 
                                                                     keep_prob: 1.0})
            softmax_logit_vals_arr.append(softmax_logit_vals)
            
            start_idx += batch_size
        
    final_softmax_logit_vals = np.concatenate(softmax_logit_vals_arr, axis=0)
    
    assert final_softmax_logit_vals.shape[0] == X.shape[0]
    assert final_softmax_logit_vals.shape[1] == 43
    
    return final_softmax_logit_vals
In [81]:
def get_best_softmax_logits(softmax_logits):
    softmax_logits_pd = pd.DataFrame(data={'softmax_logit': softmax_logits}, index=range(43))
    
    softmax_logits_pd.sort_values(by='softmax_logit', ascending=False, inplace=True)
    
    return softmax_logits_pd.index[0]
In [82]:
def vote_ensemble_results(y_hat_arr):
    er_arr = []
    
    for i in range(y_hat_arr[0].shape[0]):
        curr_results = []
        for j in range(len(y_hat_arr)):
            curr_results.append(y_hat_arr[j][i])
            
        mode_result = Counter(curr_results).most_common(1)
        mode_val = mode_result[0][0]
        er_arr.append(mode_val)     
    
    er = np.array(er_arr).reshape((-1))  
    assert er.shape[0] == y_hat_arr[0].shape[0]     
                                               
    return er     
In [83]:
def predict_msl_ensemble_model(X, model_prefix, n, batch_size):
    softmax_logits_arr = [predict_model_softmax_logits(X, '%s_%d' % (model_prefix, i), 
                                                       batch_size) for i in tqdm(range(n))]
    softmax_logits_np = np.stack(softmax_logits_arr)
    
    assert softmax_logits_np.shape[0] == n
    
    mean_softmax_logits_np = np.mean(softmax_logits_np, axis=0)
    
    assert mean_softmax_logits_np.shape[0] == X.shape[0]
    assert mean_softmax_logits_np.shape[1] == 43
    
    best_softmax_logits_arr = []
    for i in tqdm(range(mean_softmax_logits_np.shape[0])):
        curr_mean_softmax_logits_np = mean_softmax_logits_np[i]
        
        best_softmax_logits_arr.append(get_best_softmax_logits(curr_mean_softmax_logits_np))
        
    return np.array(best_softmax_logits_arr)
Ensemble Methods

Two different ensemble methods were used:

  1. Voting
    • The predicted label from each model counts as one vote.
    • The label with the most votes is the predicted label.
  2. Mean Softmax Logits
    • For each model, all of the softmax logits for an image were calculated.
    • From the softmax logits for all models, the means of each softmax logit were taken.
    • The logit with the max mean softmax is the predicted label.
Voting
In [84]:
# Train accuracy
y_ensembled_train_hat = vote_ensemble_results(y_ensemble_train_hat_arr)
train_accuracy = get_individual_accuracy(y_train_preprocessed, y_ensembled_train_hat)

# Validation Accuracy
y_ensembled_valid_hat = vote_ensemble_results(y_ensemble_valid_hat_arr)
valid_accuracy = get_individual_accuracy(y_valid_preprocessed, y_ensembled_valid_hat)

# Test Accuracy
y_ensembled_test_hat = vote_ensemble_results(y_ensemble_test_hat_arr)
test_accuracy = get_individual_accuracy(y_test_preprocessed, y_ensembled_test_hat)
In [85]:
display_accuracy_table(train_accuracy, valid_accuracy, test_accuracy)
Type	Accuracy
================
train	1.000
valid	0.993
test	0.984
In [86]:
display_single_accuracy_barplot(train_accuracy, valid_accuracy, test_accuracy,
                               'Voting Ensemble Model Accuracies')
Mean Softmax Logits
In [87]:
# Train accuracy
y_ensembled_train_hat = predict_msl_ensemble_model(X_train_preprocessed, 'model', 10, BATCH_SIZE)
train_accuracy = get_individual_accuracy(y_train_preprocessed, y_ensembled_train_hat)

# Validation Accuracy
y_ensembled_valid_hat = predict_msl_ensemble_model(X_valid_preprocessed, 'model', 10, BATCH_SIZE)
valid_accuracy = get_individual_accuracy(y_valid_preprocessed, y_ensembled_valid_hat)

# Test Accuracy
y_ensembled_test_hat = predict_msl_ensemble_model(X_test_preprocessed, 'model', 10, BATCH_SIZE)
test_accuracy = get_individual_accuracy(y_test_preprocessed, y_ensembled_test_hat)
100%|██████████| 10/10 [00:37<00:00,  3.81s/it]
100%|██████████| 34799/34799 [00:28<00:00, 1209.36it/s]
100%|██████████| 10/10 [00:16<00:00,  1.72s/it]
100%|██████████| 4410/4410 [00:03<00:00, 1197.15it/s]
100%|██████████| 10/10 [00:24<00:00,  2.50s/it]
100%|██████████| 12630/12630 [00:10<00:00, 1262.89it/s]
In [88]:
display_accuracy_table(train_accuracy, valid_accuracy, test_accuracy)
Type	Accuracy
================
train	1.000
valid	0.992
test	0.984
In [89]:
display_single_accuracy_barplot(train_accuracy, valid_accuracy, test_accuracy,
                               'Mean Softmax Logits Model Accuracies')

For the most part, the accuracy of both models are the same. Both have the same test accuracy of 99.0% The Mean Softmax Logits model has a slightly slower a validation accuracy of 98.9%.

Incorrect Labels

Validation

In [90]:
display_incorrect_labels(y_valid_preprocessed, y_ensembled_valid_hat)
#     Label                                              Count
==============================================================
21    Double curve                                          10
24    Road narrows on the right                             5
20    Dangerous curve to the right                          5
16    Vehicles over 3.5 metric tons prohibited              3
1     Speed limit (30km/h)                                  3
41    End of no passing                                     3
18    General caution                                       1
9     No passing                                            1
23    Slippery road                                         1
27    Pedestrians                                           1

Test

In [91]:
display_incorrect_labels(y_test_preprocessed, y_ensembled_test_hat)
#     Label                                              Count
==============================================================
21    Double curve                                          34
26    Traffic signals                                       21
3     Speed limit (60km/h)                                  20
18    General caution                                       19
11    Right-of-way at the next intersection                 14
41    End of no passing                                     11
6     End of speed limit (80km/h)                           9
12    Priority road                                         9
24    Road narrows on the right                             8
30    Beware of ice/snow                                    8

Step 3: Test a Model on New Images

To give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Load and Output the Images

In [92]:
### Load the images and plot them here.
### Feel free to use as many code cells as needed.
In [93]:
def read_images(img_dir, csv_file, csv_delimiter=';'):
    meta_pd = pd.read_csv('%s/%s' % (img_dir, csv_file), delimiter=csv_delimiter)
    
    image_arr = []
    
    for index, row in meta_pd.iterrows():
        filename = '%s/%s' % (img_dir, row['Filename'])
        
        pil_image = Image.open(filename)
        image = np.array(pil_image)
        image_arr.append(image)
        
    return image_arr, meta_pd
In [94]:
def crop_resize_images(image_arr, meta_pd, resize_dim):
    cropped_resized_images = []
    
    for index, row in meta_pd.iterrows():
        image = image_arr[index]
        assert image.shape[0] == row['Height'], "image.shape[0] = %d, row['Height'] = %d" % (image.shape[0], row['Height'])
        assert image.shape[1] == row['Width'], "image.shape[1] = %d, row['Width'] = %d" % (image.shape[1], row['Width'])
        
        cropped_image = image[row['Roi.Y1']:row['Roi.Y2'], row['Roi.X1']:row['Roi.X2']]   
        resized_image = resize(cropped_image, resize_dim, mode='constant')
        
        cropped_resized_images.append(resized_image)
        
    return np.stack(cropped_resized_images)
In [95]:
def make_image_dataset(main_img_dir, resize_dim=(32, 32, 3), csv_delimiter=';', has_subdirs=True):
    feature_arr = []
    label_arr = []
    sizes = []
    coords = []
    
    if has_subdirs:
        for curr_dir in tqdm([x for x in os.listdir(main_img_dir) if not x.startswith('.') and x != 'annotations']):
            csv_file = [x for x in os.listdir('%s/%s' % (main_img_dir, curr_dir)) if x.endswith('.csv')][0]
            image_arr, meta_pd = read_images('%s/%s' % (main_img_dir, curr_dir), csv_file, csv_delimiter=csv_delimiter)
                                          
            cr_images = crop_resize_images(image_arr, meta_pd, resize_dim)
            feature_arr.append(cr_images)
        
            label_arr += [int(curr_dir)] * cr_images.shape[0]
            sizes += [(row['Width'], row['Height']) for _, row in meta_pd.iterrows()]
            coords += [(row['Roi.X1'], row['Roi.Y1'], row['Roi.X2'], row['Roi.Y2']) for _, row in meta_pd.iterrows()]
    else:
        csv_file = [x for x in os.listdir(main_img_dir) if x.endswith('.csv')][0]
        image_arr, meta_pd = read_images(main_img_dir, csv_file, csv_delimiter=csv_delimiter)
                                          
        cr_images = crop_resize_images(image_arr, meta_pd, resize_dim)
        feature_arr.append(cr_images)
        
        # since it's not in a subdir, set it to an unknown label
        label_arr += meta_pd['ClassId'].tolist()
        sizes += [(row['Width'], row['Height']) for _, row in meta_pd.iterrows()]
        coords += [(row['Roi.X1'], row['Roi.Y1'], row['Roi.X2'], row['Roi.Y2']) for _, row in meta_pd.iterrows()]        
          
    features = (255.0 * np.concatenate(feature_arr, axis=0)).astype(np.uint8)
    labels = np.array(label_arr)
    
    assert features.shape[0] == labels.shape[0]
    assert labels.shape[0] == len(sizes)
    assert len(sizes) == len(coords)
    
    return {
        'features': features,
        'labels': labels,
        'sizes': sizes,
        'coords': coords
    }                                                                    
In [96]:
extra_images_dir = 'extra'
extra_image_dataset = make_image_dataset(extra_images_dir, resize_dim=(32, 32, 3), csv_delimiter="\t", has_subdirs=False)
In [97]:
X_extra, y_extra = extra_image_dataset['features'], extra_image_dataset['labels']
In [98]:
display_X_y_images(X_extra, y_extra, signnames_pd, 0, 10)
In [99]:
X_extra_preprocessed = preprocess_images(X_extra)
y_extra_preprocessed = y_extra
100%|██████████| 10/10 [00:00<00:00, 64.08it/s]
100%|██████████| 4/4 [00:00<00:00, 4386.20it/s]

Predict the Sign Type for Each Image

In [100]:
### Run the predictions here and use the model to output the prediction for each image.
### Make sure to pre-process the images with the same pre-processing pipeline used earlier.
### Feel free to use as many code cells as needed.
In [101]:
def display_predicted_labels(y, y_hat):
    correct_cnt = 0 
    
    header_str = pad_str("label", 40)
    header_str += "predicted label"
    print(header_str)
    
    print("==============================================================")
    for i, curr_y_hat in enumerate(y_hat):
        row_str = '(%d) %s' % (y[i], signname_val_to_name(signnames_pd, y[i]))
        row_str = pad_str(row_str, 40)
        
        row_str += '(%d) %s' % (curr_y_hat, signname_val_to_name(signnames_pd, curr_y_hat))
        
        print(row_str)
        
        if y[i] == curr_y_hat:
            correct_cnt += 1
           
    perc_correct = correct_cnt / len(y)    
    print()
    print("Percent Correct: %.3f" % perc_correct)

Voting

In [102]:
n = 10
y_extra_hat_arr = predict_ensemble_models(X_extra_preprocessed, 'model', 
                                            n, batch_size=BATCH_SIZE)

y_extra_hat = vote_ensemble_results(y_extra_hat_arr)
voting_accuracy = get_individual_accuracy(y_extra_preprocessed, y_extra_hat)
100%|██████████| 10/10 [00:12<00:00,  1.26s/it]
In [103]:
display_predicted_labels(y_extra_preprocessed, y_extra_hat)
label                                   predicted label
==============================================================
(1) Speed limit (30km/h)                (1) Speed limit (30km/h)
(27) Pedestrians                        (27) Pedestrians
(28) Children crossing                  (28) Children crossing
(40) Roundabout mandatory               (40) Roundabout mandatory
(35) Ahead only                         (35) Ahead only
(23) Slippery road                      (23) Slippery road
(17) No entry                           (17) No entry
(24) Road narrows on the right          (24) Road narrows on the right
(29) Bicycles crossing                  (28) Children crossing
(18) General caution                    (18) General caution

Percent Correct: 0.900

Mean Softmax Logits

In [104]:
n = 10

y_extra_hat = predict_msl_ensemble_model(X_extra_preprocessed, 'model', 
                                     10, BATCH_SIZE)
msl_accuracy = get_individual_accuracy(y_extra_preprocessed, y_extra_hat)
100%|██████████| 10/10 [00:19<00:00,  1.96s/it]
100%|██████████| 10/10 [00:00<00:00, 1097.44it/s]
In [105]:
display_predicted_labels(y_extra_preprocessed, y_extra_hat)
label                                   predicted label
==============================================================
(1) Speed limit (30km/h)                (1) Speed limit (30km/h)
(27) Pedestrians                        (27) Pedestrians
(28) Children crossing                  (28) Children crossing
(40) Roundabout mandatory               (40) Roundabout mandatory
(35) Ahead only                         (35) Ahead only
(23) Slippery road                      (23) Slippery road
(17) No entry                           (17) No entry
(24) Road narrows on the right          (24) Road narrows on the right
(29) Bicycles crossing                  (28) Children crossing
(18) General caution                    (18) General caution

Percent Correct: 0.900

Valuation of Mislabeled Image

In [107]:
plt.figure(figsize=(2,2))
plt.title('Extra - Bicycles crossing')
plt.imshow(X_extra[8])
plt.show()

print('Sample "Bicycles crossing"')
X_pedestrians = filter_images_by_label(29, X=X_valid, y=y_valid)
display_X_y_images(X_pedestrians, [29]*5, signnames_pd=signnames_pd, 
                   start_idx=0, end_idx=5)

print('Sample "Children crossing"')
X_valid_right_of_way = filter_images_by_label(28, X=X_valid, y=y_valid)
display_X_y_images(X_valid_right_of_way, [28]*5, signnames_pd=signnames_pd, 
                   start_idx=0, end_idx=5)
Sample "Bicycles crossing"
Sample "Children crossing"

Analyze Performance

In [108]:
### Calculate the accuracy for these 5 new images. 
### For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate on these new images.
In [109]:
print("Model Type             \tAccuracy")
print("================================")
print("Voting             \t%.3f" % voting_accuracy)
print("Mean Softmax Logits\t%.3f" % msl_accuracy)
Model Type             	Accuracy
================================
Voting             	0.900
Mean Softmax Logits	0.900

The Mean Softmax Logits model had a higher accuracy compared to the Voting model but there were only 10 images. The sample size is probably too small to definitively say that the Mean Softmax Logits is the better model.

Output Top 5 Softmax Probabilities For Each Image Found on the Web

For each of the new images, print out the model's softmax probabilities to show the certainty of the model's predictions (limit the output to the top 5 probabilities for each image). tf.nn.top_k could prove helpful here.

The example below demonstrates how tf.nn.top_k can be used to find the top k predictions for each image.

tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.

Take this numpy array as an example. The values in the array represent predictions. The array contains softmax probabilities for five candidate images with six possible classes. tf.nn.top_k is used to choose the three classes with the highest probability:

# (5, 6) array
a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
         0.12789202],
       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
         0.15899337],
       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
         0.23892179],
       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
         0.16505091],
       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
         0.09155967]])

Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:

TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
       [ 0.28086119,  0.27569815,  0.18063401],
       [ 0.26076848,  0.23892179,  0.23664738],
       [ 0.29198961,  0.26234032,  0.16505091],
       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
       [0, 1, 4],
       [0, 5, 1],
       [1, 3, 5],
       [1, 4, 3]], dtype=int32))

Looking just at the first row we get [ 0.34763842, 0.24879643, 0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.

In [110]:
### Print out the top five softmax probabilities for the predictions on the German traffic sign images found on the web. 
### Feel free to use as many code cells as needed.
In [111]:
def predict_model_top_logits(X, model_name, batch_size):
    if X.shape[0] < batch_size:
        batch_size = X.shape[0]
        
    model_dir = '%s/models/%s' % (data_dir, model_name)
    saver = tf.train.Saver()
    
    logit_vals_arr = []
    label_vals_arr = []
    
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())  
        saver.restore(sess, tf.train.latest_checkpoint(model_dir))
        
        top_logits = tf.nn.top_k(tf.nn.softmax(logits), k=5, name='top_logits')
        
        start_idx = 0
        end_idx = 0
        while start_idx < X.shape[0]:
            end_idx += batch_size
            end_idx = min(end_idx, X.shape[0])
            logit_vals, label_vals = sess.run(top_logits, feed_dict={x: X[start_idx:end_idx], 
                                                                     keep_prob: 1.0})
            logit_vals_arr.append(logit_vals)
            label_vals_arr.append(label_vals)
            
            start_idx += batch_size
        
    final_logit_vals = np.concatenate(logit_vals_arr, axis=0)
    final_label_vals = np.concatenate(label_vals_arr, axis=0)
    
    assert final_logit_vals.shape[0] == X.shape[0]
    assert final_label_vals.shape[0] == X.shape[0]
    
    return final_logit_vals, final_label_vals
In [112]:
def predict_voting_ensemble_models_top_logits(X, model_prefix, n, batch_size):
    results = []
    for i in tqdm(range(n)):
        model_name = '%s_%d' % (model_prefix, i)
        
        results.append(predict_model_top_logits(X, model_name, batch_size=batch_size))
    
    # (num_models, logit_labels=2, num_images, n)
    return results
In [113]:
top_logits_result_arr = predict_voting_ensemble_models_top_logits(X_extra_preprocessed, 
                                                                  'model', 10, BATCH_SIZE)
100%|██████████| 10/10 [00:20<00:00,  2.07s/it]

Here are the top logits for each of the individual models.

In [114]:
for i in range(y_extra.shape[0]): 
    print("%d: %s" % (i, signname_val_to_name(signnames_pd, y_extra[i])))
    
    for j in range(len(top_logits_result_arr)):
        print("\tModel %d" % j)
        logit_vals, label_vals = top_logits_result_arr[j]
        curr_logit_vals = logit_vals[i]
        curr_label_vals = label_vals[i]
        for k in range(curr_logit_vals.shape[0]):
            print("\t%.4f\t%s" % (curr_logit_vals[k], 
                                  signname_val_to_name(signnames_pd, curr_label_vals[k])))
              
        print()
        
    print("==========================================")
0: Speed limit (30km/h)
	Model 0
	1.0000	Speed limit (30km/h)
	0.0000	Turn right ahead
	0.0000	Speed limit (20km/h)
	0.0000	End of speed limit (80km/h)
	0.0000	Children crossing

	Model 1
	1.0000	Speed limit (30km/h)
	0.0000	Speed limit (50km/h)
	0.0000	Stop
	0.0000	End of all speed and passing limits
	0.0000	End of speed limit (80km/h)

	Model 2
	1.0000	Speed limit (30km/h)
	0.0000	End of speed limit (80km/h)
	0.0000	Keep left
	0.0000	Speed limit (50km/h)
	0.0000	Stop

	Model 3
	1.0000	Speed limit (30km/h)
	0.0000	Yield
	0.0000	Keep left
	0.0000	Speed limit (50km/h)
	0.0000	Bumpy road

	Model 4
	1.0000	Speed limit (30km/h)
	0.0000	Speed limit (20km/h)
	0.0000	Speed limit (50km/h)
	0.0000	Road work
	0.0000	Speed limit (70km/h)

	Model 5
	0.6552	Go straight or left
	0.2059	Keep left
	0.0609	General caution
	0.0145	End of speed limit (80km/h)
	0.0139	Speed limit (60km/h)

	Model 6
	1.0000	Speed limit (30km/h)
	0.0000	Speed limit (20km/h)
	0.0000	Stop
	0.0000	Speed limit (70km/h)
	0.0000	Speed limit (50km/h)

	Model 7
	1.0000	Speed limit (30km/h)
	0.0000	Speed limit (70km/h)
	0.0000	Stop
	0.0000	Speed limit (20km/h)
	0.0000	Yield

	Model 8
	1.0000	Speed limit (30km/h)
	0.0000	Speed limit (50km/h)
	0.0000	Bicycles crossing
	0.0000	Stop
	0.0000	Speed limit (80km/h)

	Model 9
	0.6183	Speed limit (30km/h)
	0.3670	End of speed limit (80km/h)
	0.0141	Children crossing
	0.0003	Beware of ice/snow
	0.0001	Bicycles crossing

==========================================
1: Pedestrians
	Model 0
	0.5902	Traffic signals
	0.4083	Pedestrians
	0.0015	Children crossing
	0.0001	Road narrows on the right
	0.0000	Dangerous curve to the left

	Model 1
	0.8128	Children crossing
	0.1736	Road work
	0.0026	Stop
	0.0023	End of all speed and passing limits
	0.0019	Vehicles over 3.5 metric tons prohibited

	Model 2
	0.8809	General caution
	0.0442	Road narrows on the right
	0.0296	Pedestrians
	0.0270	Children crossing
	0.0102	Bicycles crossing

	Model 3
	0.8570	Road narrows on the right
	0.1427	Pedestrians
	0.0002	General caution
	0.0001	Beware of ice/snow
	0.0000	Children crossing

	Model 4
	0.9813	Pedestrians
	0.0134	Traffic signals
	0.0049	Road narrows on the right
	0.0003	Children crossing
	0.0000	General caution

	Model 5
	0.8023	Road narrows on the right
	0.1831	Beware of ice/snow
	0.0124	Pedestrians
	0.0011	Right-of-way at the next intersection
	0.0009	Double curve

	Model 6
	0.9866	Pedestrians
	0.0094	Traffic signals
	0.0035	Right-of-way at the next intersection
	0.0003	Go straight or right
	0.0002	General caution

	Model 7
	0.4638	Pedestrians
	0.4542	Children crossing
	0.0523	Road narrows on the right
	0.0123	Stop
	0.0101	Traffic signals

	Model 8
	0.9196	Pedestrians
	0.0803	Right-of-way at the next intersection
	0.0001	Traffic signals
	0.0000	Road work
	0.0000	General caution

	Model 9
	0.9236	Road work
	0.0759	General caution
	0.0002	Road narrows on the right
	0.0002	Beware of ice/snow
	0.0001	Turn left ahead

==========================================
2: Children crossing
	Model 0
	0.9999	Children crossing
	0.0000	Beware of ice/snow
	0.0000	Dangerous curve to the right
	0.0000	Right-of-way at the next intersection
	0.0000	End of no passing

	Model 1
	0.8630	Bicycles crossing
	0.0768	Children crossing
	0.0540	Beware of ice/snow
	0.0035	Speed limit (60km/h)
	0.0013	Road work

	Model 2
	0.7396	Children crossing
	0.2598	Beware of ice/snow
	0.0006	Dangerous curve to the right
	0.0000	Go straight or left
	0.0000	No entry

	Model 3
	0.9893	Children crossing
	0.0091	Dangerous curve to the right
	0.0006	Bicycles crossing
	0.0003	Pedestrians
	0.0002	End of no passing by vehicles over 3.5 metric tons

	Model 4
	0.8609	Bicycles crossing
	0.0957	Children crossing
	0.0272	Beware of ice/snow
	0.0107	Slippery road
	0.0034	Road narrows on the right

	Model 5
	0.5158	Dangerous curve to the right
	0.4336	Slippery road
	0.0220	Children crossing
	0.0204	Bicycles crossing
	0.0029	Vehicles over 3.5 metric tons prohibited

	Model 6
	0.9970	Children crossing
	0.0011	Road narrows on the right
	0.0008	Ahead only
	0.0007	Beware of ice/snow
	0.0002	Dangerous curve to the right

	Model 7
	0.7641	Road work
	0.1145	Children crossing
	0.0796	Speed limit (30km/h)
	0.0108	End of no passing
	0.0107	Road narrows on the right

	Model 8
	0.9842	Children crossing
	0.0140	Road narrows on the right
	0.0007	Beware of ice/snow
	0.0006	Bumpy road
	0.0004	Bicycles crossing

	Model 9
	0.3935	Children crossing
	0.3130	Beware of ice/snow
	0.2309	Bicycles crossing
	0.0266	Bumpy road
	0.0077	Speed limit (60km/h)

==========================================
3: Roundabout mandatory
	Model 0
	0.9929	Roundabout mandatory
	0.0040	End of no passing by vehicles over 3.5 metric tons
	0.0022	Keep right
	0.0008	Speed limit (100km/h)
	0.0000	Keep left

	Model 1
	0.9999	Keep right
	0.0001	Roundabout mandatory
	0.0000	Go straight or right
	0.0000	Turn left ahead
	0.0000	Beware of ice/snow

	Model 2
	0.7269	Roundabout mandatory
	0.2341	Keep left
	0.0352	End of speed limit (80km/h)
	0.0033	Keep right
	0.0002	Speed limit (30km/h)

	Model 3
	0.9914	Roundabout mandatory
	0.0086	Go straight or left
	0.0000	Keep right
	0.0000	Turn right ahead
	0.0000	Keep left

	Model 4
	0.9976	Roundabout mandatory
	0.0024	Go straight or left
	0.0000	Priority road
	0.0000	Keep left
	0.0000	Turn left ahead

	Model 5
	1.0000	Roundabout mandatory
	0.0000	Go straight or left
	0.0000	Vehicles over 3.5 metric tons prohibited
	0.0000	No passing for vehicles over 3.5 metric tons
	0.0000	Speed limit (80km/h)

	Model 6
	0.9997	Roundabout mandatory
	0.0002	Keep right
	0.0001	Go straight or right
	0.0000	Priority road
	0.0000	Go straight or left

	Model 7
	0.9962	Roundabout mandatory
	0.0030	Go straight or left
	0.0006	End of no passing
	0.0001	Turn right ahead
	0.0000	No vehicles

	Model 8
	1.0000	Roundabout mandatory
	0.0000	Go straight or left
	0.0000	Keep left
	0.0000	Keep right
	0.0000	Turn right ahead

	Model 9
	0.9572	Roundabout mandatory
	0.0316	Keep right
	0.0107	Keep left
	0.0003	No passing for vehicles over 3.5 metric tons
	0.0001	Go straight or right

==========================================
4: Ahead only
	Model 0
	1.0000	Ahead only
	0.0000	Go straight or right
	0.0000	Turn left ahead
	0.0000	General caution
	0.0000	No vehicles

	Model 1
	0.9729	Ahead only
	0.0237	Go straight or right
	0.0015	Turn right ahead
	0.0014	Go straight or left
	0.0002	Turn left ahead

	Model 2
	0.9999	Ahead only
	0.0001	Roundabout mandatory
	0.0000	Go straight or left
	0.0000	Turn left ahead
	0.0000	Go straight or right

	Model 3
	1.0000	Ahead only
	0.0000	Go straight or left
	0.0000	Priority road
	0.0000	Speed limit (60km/h)
	0.0000	No passing

	Model 4
	0.9841	Go straight or left
	0.0155	Ahead only
	0.0003	Roundabout mandatory
	0.0001	Priority road
	0.0000	Keep right

	Model 5
	0.9999	Ahead only
	0.0001	Go straight or left
	0.0000	Priority road
	0.0000	Roundabout mandatory
	0.0000	Turn right ahead

	Model 6
	1.0000	Ahead only
	0.0000	Turn left ahead
	0.0000	Go straight or right
	0.0000	Right-of-way at the next intersection
	0.0000	Turn right ahead

	Model 7
	1.0000	Ahead only
	0.0000	End of no passing
	0.0000	Turn left ahead
	0.0000	Children crossing
	0.0000	Go straight or right

	Model 8
	1.0000	Ahead only
	0.0000	Yield
	0.0000	Go straight or left
	0.0000	No vehicles
	0.0000	Road narrows on the right

	Model 9
	0.9997	Ahead only
	0.0002	Go straight or right
	0.0000	Go straight or left
	0.0000	Turn left ahead
	0.0000	Roundabout mandatory

==========================================
5: Slippery road
	Model 0
	1.0000	Slippery road
	0.0000	Beware of ice/snow
	0.0000	Dangerous curve to the left
	0.0000	Children crossing
	0.0000	Dangerous curve to the right

	Model 1
	0.9738	Children crossing
	0.0171	Bicycles crossing
	0.0032	Beware of ice/snow
	0.0023	Road work
	0.0022	Slippery road

	Model 2
	0.9993	Slippery road
	0.0004	Dangerous curve to the right
	0.0002	Bicycles crossing
	0.0000	Bumpy road
	0.0000	Beware of ice/snow

	Model 3
	0.6809	Road narrows on the right
	0.2890	Dangerous curve to the left
	0.0205	Pedestrians
	0.0075	Double curve
	0.0012	Slippery road

	Model 4
	0.7902	Slippery road
	0.1838	Beware of ice/snow
	0.0161	Dangerous curve to the right
	0.0056	Bicycles crossing
	0.0025	Dangerous curve to the left

	Model 5
	1.0000	Slippery road
	0.0000	Vehicles over 3.5 metric tons prohibited
	0.0000	Wild animals crossing
	0.0000	Dangerous curve to the right
	0.0000	Dangerous curve to the left

	Model 6
	1.0000	Slippery road
	0.0000	Dangerous curve to the right
	0.0000	Dangerous curve to the left
	0.0000	Vehicles over 3.5 metric tons prohibited
	0.0000	Speed limit (80km/h)

	Model 7
	0.9927	Slippery road
	0.0055	Bicycles crossing
	0.0007	Dangerous curve to the right
	0.0007	Pedestrians
	0.0003	Beware of ice/snow

	Model 8
	0.9661	Slippery road
	0.0273	Bicycles crossing
	0.0066	Children crossing
	0.0000	Beware of ice/snow
	0.0000	Speed limit (60km/h)

	Model 9
	0.7509	Bicycles crossing
	0.2316	Bumpy road
	0.0173	Speed limit (60km/h)
	0.0002	Slippery road
	0.0000	Wild animals crossing

==========================================
6: No entry
	Model 0
	1.0000	No entry
	0.0000	Stop
	0.0000	Double curve
	0.0000	Slippery road
	0.0000	Speed limit (50km/h)

	Model 1
	1.0000	No entry
	0.0000	Stop
	0.0000	Dangerous curve to the left
	0.0000	Pedestrians
	0.0000	No passing

	Model 2
	1.0000	No entry
	0.0000	Stop
	0.0000	Speed limit (60km/h)
	0.0000	Go straight or left
	0.0000	Children crossing

	Model 3
	1.0000	No entry
	0.0000	Stop
	0.0000	Speed limit (100km/h)
	0.0000	Speed limit (120km/h)
	0.0000	Dangerous curve to the right

	Model 4
	1.0000	No entry
	0.0000	No passing
	0.0000	Speed limit (60km/h)
	0.0000	Speed limit (120km/h)
	0.0000	Dangerous curve to the right

	Model 5
	1.0000	No entry
	0.0000	Traffic signals
	0.0000	Bicycles crossing
	0.0000	No passing
	0.0000	Speed limit (80km/h)

	Model 6
	1.0000	No entry
	0.0000	Stop
	0.0000	Traffic signals
	0.0000	Yield
	0.0000	Speed limit (30km/h)

	Model 7
	1.0000	No entry
	0.0000	Stop
	0.0000	Children crossing
	0.0000	Speed limit (50km/h)
	0.0000	Bicycles crossing

	Model 8
	1.0000	No entry
	0.0000	Stop
	0.0000	No passing
	0.0000	Speed limit (50km/h)
	0.0000	Speed limit (60km/h)

	Model 9
	1.0000	No entry
	0.0000	No passing
	0.0000	Stop
	0.0000	Speed limit (120km/h)
	0.0000	End of no passing

==========================================
7: Road narrows on the right
	Model 0
	0.9739	Road narrows on the right
	0.0193	Pedestrians
	0.0067	Dangerous curve to the left
	0.0000	Traffic signals
	0.0000	Children crossing

	Model 1
	0.9481	Children crossing
	0.0215	Right-of-way at the next intersection
	0.0213	Road work
	0.0035	Beware of ice/snow
	0.0032	Bicycles crossing

	Model 2
	1.0000	Road narrows on the right
	0.0000	Beware of ice/snow
	0.0000	General caution
	0.0000	Children crossing
	0.0000	Bicycles crossing

	Model 3
	0.9431	Double curve
	0.0512	Road narrows on the right
	0.0056	Road work
	0.0000	Wild animals crossing
	0.0000	Beware of ice/snow

	Model 4
	1.0000	Road narrows on the right
	0.0000	Bicycles crossing
	0.0000	Pedestrians
	0.0000	Traffic signals
	0.0000	Bumpy road

	Model 5
	1.0000	Road narrows on the right
	0.0000	Dangerous curve to the left
	0.0000	Double curve
	0.0000	Slippery road
	0.0000	Speed limit (20km/h)

	Model 6
	0.7980	Road narrows on the right
	0.1949	Ahead only
	0.0070	Turn left ahead
	0.0001	Beware of ice/snow
	0.0000	Right-of-way at the next intersection

	Model 7
	0.8263	Road narrows on the right
	0.1172	Pedestrians
	0.0508	Bicycles crossing
	0.0031	Children crossing
	0.0010	Double curve

	Model 8
	0.9995	Road narrows on the right
	0.0004	Pedestrians
	0.0000	Road work
	0.0000	Yield
	0.0000	Children crossing

	Model 9
	0.9999	Road work
	0.0001	Beware of ice/snow
	0.0000	Road narrows on the right
	0.0000	Double curve
	0.0000	Speed limit (30km/h)

==========================================
8: Bicycles crossing
	Model 0
	0.5893	Speed limit (60km/h)
	0.3413	Children crossing
	0.0269	Vehicles over 3.5 metric tons prohibited
	0.0192	Stop
	0.0119	End of no passing

	Model 1
	0.9540	Children crossing
	0.0230	Speed limit (30km/h)
	0.0175	Stop
	0.0025	Vehicles over 3.5 metric tons prohibited
	0.0016	Bicycles crossing

	Model 2
	0.8296	Children crossing
	0.1643	Beware of ice/snow
	0.0020	End of no passing
	0.0016	Dangerous curve to the right
	0.0010	Slippery road

	Model 3
	0.3514	Children crossing
	0.2844	End of no passing
	0.1659	End of all speed and passing limits
	0.1166	Traffic signals
	0.0277	Dangerous curve to the right

	Model 4
	0.9967	Children crossing
	0.0017	Speed limit (120km/h)
	0.0009	Stop
	0.0004	Traffic signals
	0.0001	No entry

	Model 5
	0.3697	Speed limit (80km/h)
	0.3445	Speed limit (60km/h)
	0.1371	Speed limit (50km/h)
	0.0375	Bicycles crossing
	0.0295	Go straight or left

	Model 6
	0.3654	Children crossing
	0.1890	Speed limit (60km/h)
	0.1670	Bicycles crossing
	0.1013	No passing
	0.0648	Speed limit (80km/h)

	Model 7
	0.4634	Children crossing
	0.4317	Stop
	0.0681	Bicycles crossing
	0.0094	No entry
	0.0071	Yield

	Model 8
	0.8094	Stop
	0.1340	Speed limit (20km/h)
	0.0292	Speed limit (60km/h)
	0.0190	Bicycles crossing
	0.0078	Children crossing

	Model 9
	0.9218	Stop
	0.0642	Children crossing
	0.0032	Bicycles crossing
	0.0021	Slippery road
	0.0019	End of speed limit (80km/h)

==========================================
9: General caution
	Model 0
	1.0000	General caution
	0.0000	Road narrows on the right
	0.0000	Road work
	0.0000	Traffic signals
	0.0000	Pedestrians

	Model 1
	1.0000	General caution
	0.0000	Speed limit (20km/h)
	0.0000	Traffic signals
	0.0000	Dangerous curve to the right
	0.0000	Turn left ahead

	Model 2
	1.0000	General caution
	0.0000	Children crossing
	0.0000	Road narrows on the right
	0.0000	End of all speed and passing limits
	0.0000	Stop

	Model 3
	1.0000	General caution
	0.0000	Road work
	0.0000	Road narrows on the right
	0.0000	Yield
	0.0000	Turn left ahead

	Model 4
	1.0000	General caution
	0.0000	Right-of-way at the next intersection
	0.0000	Ahead only
	0.0000	Road work
	0.0000	Pedestrians

	Model 5
	1.0000	General caution
	0.0000	Pedestrians
	0.0000	Traffic signals
	0.0000	Right-of-way at the next intersection
	0.0000	Children crossing

	Model 6
	1.0000	General caution
	0.0000	Turn right ahead
	0.0000	Ahead only
	0.0000	Stop
	0.0000	Children crossing

	Model 7
	1.0000	General caution
	0.0000	Speed limit (70km/h)
	0.0000	Turn right ahead
	0.0000	Stop
	0.0000	Vehicles over 3.5 metric tons prohibited

	Model 8
	1.0000	General caution
	0.0000	Pedestrians
	0.0000	End of speed limit (80km/h)
	0.0000	Children crossing
	0.0000	Traffic signals

	Model 9
	1.0000	General caution
	0.0000	Yield
	0.0000	Bumpy road
	0.0000	Road work
	0.0000	Traffic signals

==========================================
In [115]:
def display_top_logits_barchart(top_logits_result_arr, i, columns=2):
    top_logits_result_np = np.array(top_logits_result_arr)
    
    # (num_models, logit_labels=2, num_images, n)
    curr_top_logits_result_np = top_logits_result_np[:, :, i, :]

    logit_vals = curr_top_logits_result_np[:, 0, :]
    label_vals = curr_top_logits_result_np[:, 1, :]
    
    num_of_models, num_of_top_logits = logit_vals.shape

    plt.figure(figsize=(32,48))
    
    
    rows = num_of_models / columns + 1

    for i in range(num_of_models):
        curr_logit_vals = logit_vals[i]
        curr_label_vals = label_vals[i]
        
        curr_label_names = [signname_val_to_name(signnames_pd, curr_label_vals[i]) 
                            for i in range(len(curr_label_vals))]
        
        plt.subplot(rows, columns, i + 1)
        
        plt.xlim(0.0, 1.0)
        plt.barh(range(num_of_top_logits), curr_logit_vals)
        
        plt.title("Model %d" % i).set_fontsize(32)
        plt.yticks(range(num_of_top_logits), curr_label_names, fontsize = 14)
        plt.xlabel('Logit').set_fontsize(24)
        
        plt.tight_layout()
 
    plt.show()
In [116]:
for i in range(y_extra.shape[0]): 
    label_name = "%d: %s" % (i, signname_val_to_name(signnames_pd, y_extra[i]))
    
    plt.figure(figsize=(2,2))
    plt.title(label_name)
    plt.imshow(X_extra[i])
    plt.show()

    display_top_logits_barchart(top_logits_result_arr, i)

Project Writeup

Once you have completed the code implementation, document your results in a project writeup using this template as a guide. The writeup can be in a markdown or pdf file.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.


Step 4 (Optional): Visualize the Neural Network's State with Test Images

This Section is not required to complete but acts as an additional excersise for understaning the output of a neural network's weights. While neural networks can be a great learning device they are often referred to as a black box. We can understand what the weights of a neural network look like better by plotting their feature maps. After successfully training your neural network you can see what it's feature maps look like by plotting the output of the network's weight layers in response to a test stimuli image. From these plotted feature maps, it's possible to see what characteristics of an image the network finds interesting. For a sign, maybe the inner network feature maps react with high activation to the sign's boundary outline or to the contrast in the sign's painted symbol.

Provided for you below is the function code that allows you to get the visualization output of any tensorflow weight layer you want. The inputs to the function should be a stimuli image, one used during training or a new one you provided, and then the tensorflow variable name that represents the layer's state during the training process, for instance if you wanted to see what the LeNet lab's feature maps looked like for it's second convolutional layer you could enter conv2 as the tf_activation variable.

For an example of what feature map outputs look like, check out NVIDIA's results in their paper End-to-End Deep Learning for Self-Driving Cars in the section Visualization of internal CNN State. NVIDIA was able to show that their network's inner weights had high activations to road boundary lines by comparing feature maps from an image with a clear path to one without. Try experimenting with a similar test to show that your trained network's weights are looking for interesting features, whether it's looking at differences in feature maps from images with or without a sign, or even what feature maps look like in a trained network vs a completely untrained one on the same sign image.

Combined Image

Your output should look something like this (above)

In [117]:
### Visualize your network's feature maps here.
### Feel free to use as many code cells as needed.

# image_input: the test image being fed into the network to produce the feature maps
# tf_activation: should be a tf variable name used during your training procedure that represents the calculated state of a specific weight layer
# activation_min/max: can be used to view the activation contrast in more detail, by default matplot sets min and max to the actual min and max values of the output
# plt_num: used to plot out multiple different weight feature map sets on the same block, just extend the plt number for each new feature map entry

def outputFeatureMap(image_input, tf_activation, activation_min=-1, activation_max=-1,
                     plt_num=1, columns = 4, figsize=(15,15)):
    # Here make sure to preprocess your image_input in a way your network expects
    # with size, normalization, ect if needed
    # image_input =
    # Note: x should be the same name as your network's tensorflow data placeholder variable
    # If you get an error tf_activation is not defined it may be having trouble accessing the variable from inside a function
    activation = tf_activation.eval(session=sess,feed_dict={x : image_input})

    featuremaps = activation.shape[3]
    plt.figure(plt_num, figsize=figsize)
    
    rows = featuremaps / columns + 1
    for featuremap in range(featuremaps):
        plt.subplot(rows, columns, featuremap+1) # sets the number of feature maps to show on each row and column
        plt.title('FeatureMap ' + str(featuremap)) # displays the feature map number
        if activation_min != -1 & activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin =activation_min, vmax=activation_max, cmap="gray")
        elif activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmax=activation_max, cmap="gray")
        elif activation_min !=-1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin=activation_min, cmap="gray")
        else:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", cmap="gray")
            
    plt.tight_layout()
    plt.show()
In [121]:
model_name = 'single_model'    
model_dir = '%s/models/%s' % (data_dir, model_name)
saver = tf.train.Saver()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.restore(sess, tf.train.latest_checkpoint(model_dir))

    figsize_arr = [(15,15), (15,30), (15,60)]
    
    tf_activation = activations[0]
    
    for i in range(X_extra_preprocessed.shape[0]):
        label_name = "%d: %s" % (i, signname_val_to_name(signnames_pd, y_extra[i]))
        
        curr_image = X_extra[i]
        plt.imshow(curr_image)
        plt.show()
        
        image_input = X_extra_preprocessed[i].reshape((1, 32, 32, 4))
    
        outputFeatureMap(image_input, tf_activation, columns=4)
In [ ]: